Archives for 21 Jun,2021

You are browsing the site archives by date.

← Back to Blog

Managing Big Data with Informer

Madhavi Chandra, Director of Informer Product Management, shares how Informer’s Dataset architecture, particularly features in Elasticsearch, can assist in managing Big Data. 

When data is abundant, finding actionable insights can be like finding a needle in a haystack. The key to extracting insights from big data is preparing the Dataset in a meaningful manner, and then being able to interact with it efficiently. If it is cumbersome or time-consuming to obtain results, it quickly diminishes the value of the Dataset.

For many of our customers, it is common to have Datasets with 10s or 100s of millions of rows and hundreds of columns wide. It is essential that their experience interacting with such large volumes of data is fast, efficient, accurate, and provides them the business insight they need to elevate their organization.

With Informer, you can curate as much data as you desire in the Dataset and then pare down views simply by filtering. Filtering eliminates the need for multiple Datasets and supports a single source of truth that contains the entire data picture. There is no need to query your productional database each time you interact with the Dataset; in fact, users with appropriate permissions may refresh the Dataset on a schedule that is optimal for organizational needs or may refresh the Dataset on demand. In Informer, all column field choosers are efficient, filterable views allowing you to quickly navigate/find a field and focus the view by choosing the fields you want to see at the moment. You could reduce a 200-field Dataset to five necessary columns to address a particular task. Then, rinse and repeat with any set of columns that are most relevant to your task. Our intuitive and efficient column selection makes interacting with large and wide Datasets and Reports seamless, lending to fast performance when rendering in the UI.

Data inherently grows over time. To make it both flexible and efficient when refreshing a Dataset, Informer provides different options that innately support large Datasets. You can choose to replace all records, only add new records, or add new records and update existing records that have been changed. Refreshing a multi-million row Dataset is thus painless and fast!

Informer provides multiple methods for augmenting and preparing data to support large Datasets. Flow steps are actions that are taken on Query results after a query has been run, and before the Dataset is indexed. They allow you to create new fields and modify existing fields based on results from the Query or user input. The data is then indexed for interaction and retrieval. Elasticsearch Script Fields on the other hand, allow you to add new fields to your Dataset that are evaluated inside Elasticsearch each time the field is referenced. What does that really mean? You can augment your Dataset without requiring re-indexing! Think of it as a post-Elasticsearch calculated column Flow Step, providing huge performance gains for large Datasets due to fast execution of script fields by eliminating the indexing. Elasticsearch Script Fields have many use cases and specifically make it easy to compute fields that are ‘As of’ a specific date/time, e.g., computing an Age field, Late field, or Days Since field. Any such real time Elasticsearch Script Fields are computed quickly when you invoke the field without needing to refresh the Dataset leading to big time savings!

We have designed Informer so that our customers can have efficient, effective, and intuitive interactions when working with their data, no matter how large the Dataset.

Read More
← Back to Blog

New Feature – Web Datasource

Robin Lamb, Director of Informer Client Services, discusses Informer’s exciting new web connector.

The Web Datasource driver, an experimental feature in 5.4, connects Informer to web-based tools opening the door to many different types of datasources. The driver enables Informer to connect to a REST endpoint to retrieve data. This type of connection allows users to connect to data that is not installed internally in their environment. Users will be able to pull data from SaaS-based hosted solutions. For example, at Entrinsik, we have utilized the driver to connect Jira Software to Informer 5 to aid in the efficiency of our developmental process. Jira is where we keep track of all bugs and feature requests generated by customer use. See below on how we connected Infomer to Jira (Figure 1).

Figure 1: Web Datasource Connection

To create a Dataset, enter the query parameters into the Response Handler area of the GET Request (Figure 2).

Figure 2: Dataset GET Request

By connecting to our hosted Jira, we are able to extract data that tells us which feature enhancements have been asked for by the most customers or which bugs are most critical to fix. We are able prioritize our workflow by extracting data from the source. Connect the Web Datasource to one of your web-based tools and see the insights you can extract.

Read More
← Back to Blog

Data Governance: Using Field Security

Madhavi Chandra, Director of Informer Product Management, provides insight on taking advantage of field level security to achieve better data governance. 

The key to reporting and extracting insights from data is being able to trust the data and its dissemination. It boils down to the essential aspects of Data Governance – the ability to access and consume data reliably while meeting the competing demands between IT security and business users. Good Data Governance answers questions like, ‘How can I trust the data?’, ‘How can I have traceability of key business decisions?’, ‘How do I keep my data from getting unwieldy and uncontrollable?’, and ‘How can I ensure that only authorized persons can see the data or parts of it?’. In Informer 5, we bolster our approach to the latter question, addressing data level, content level and functional level security.

Informer 5 has inherently embodied Data Governance and found an effective balance between enabling self-service analysis and protecting sensitive business information. Organizational business needs require departments to share content amongst each other reliably and confidently. For many of our customers, it is a common use case to have different groups of authorized personnel accessing different sets of sensitive data. With so many possible permutations of who can see what, we wanted to provide our customers a way to control access down to not only a row level, but also a column level in a more manageable way.

Datasource sharing can be customized down to the granular field level. In the past, fields were restricted and shared as either all or nothing. Using the updated field security model, Admin users can restrict fields and then specify exactly which restricted fields to grant to a Team or User when sharing the Datasource. For example, Team A can be granted access to the restricted fields Salary and Social Security Number, while Team B can be granted access to the restricted fields Social Security Number and Birthdate. In this way, Informer 5 allows restricting fields and granting differentiated access to fields within the same Mapping. We updated the Datasource sharing dialog to facilitate an intuitive workflow when granting access to restricted fields, including use of search and Field Sets to quickly find the restricted fields across all Mappings to share.

In order to facilitate easy navigation of fields, Informer 5.4 introduces Field Sets – an easy way to organize fields into common groups. Each field can be assigned to a single set, and the list of all fields can be easily filtered by Field Set. Now, it is possible to quickly find all of the fields that comprise a common theme – a given Field Set.

The new security is backwards compatible with the way security works in previous versions. You will have the option to deploy the new security model to your entire instance, or on a Dataset-by-Dataset basis.

Informer 5 security draws on our experience with customer security needs and understanding the best practices for security and Data Governance. We emphasize a robust security structure while providing ease of setup and maintenance as can be seen with Informer 5.4.

Read More