Exploring Stellar Wonders: Unraveling the Enigma with GridDB¶

Introduction¶

All of us have grown up in fascination of the sky and stars at night. The mere sight of a vast, twinkling canvas above us ignites a sense of wonder and curiosity. As we gaze at the celestial tapestry, our imaginations soar, pondering the mysteries of the universe. The stars, like ancient storytellers, hold secrets and tales of cosmic proportions. They have inspired countless myths, guided explorers across vast oceans, and propelled humanity's quest for knowledge and understanding. From the timeless beauty of the Milky Way to the fiery brilliance of distant supernovae, the sky beckons us to delve deeper into its mind-boggling wonders. It is in this awe-inspiring realm that GridDB, a powerful analytical tool, emerges as a guiding light, enabling us to unlock the hidden patterns and insights woven within the intricate fabric of the stars. With GridDB, we embark on a captivating journey of exploration, unraveling the enigmatic nature of the universe and painting a vivid portrait of our cosmic cousins. Let's get started!

About the Dataset¶

The Dataset is obtained from the web portal Astronomy Nexus which is owned by astronomy enthusiast and stargazer David Nash. For the purpose of this analysis, the dataset HYG 3.0 has been downloaded. Please refer to the Creative Commons Attribution-ShareAlike license of the dataset here.

Instructions to Download the Dataset¶

The dataset is in the gz format. Use a tool such as 7-Zip File Manager for this. Below are the instructions to extract the dataset from the compressed file.

Select the .gz file (hygdata_v3.csv.gz) and click the 'Extract' button.

In the screen that follows, click on 'Extract'

The file gets extracted inside a folder with the file name 'hygdata_v3.csv'.

Understanding the Dataset Abbreviations¶

The abbreviated column names represent the different properties or attributes associated with each star in the dataset.

• id: Star ID
• hip: Hipparcos ID
• hd: Henry Draper Catalog ID
• hr: Harvard Revised Catalog ID
• gl: Gliese Catalog ID
• bf: Bayer/Flamsteed designation
• proper: Proper name of the star
• ra: Right Ascension (RA) in decimal degrees or hours
• dec: Declination (Dec) in decimal degrees
• dist: Distance to the star
• pmra: Proper motion in right ascension
• pmdec: Proper motion in declination
• rv: Radial velocity
• mag: Apparent magnitude
• absmag: Absolute magnitude
• spect: Spectral type
• ci: Color index
• x, y, z: Cartesian coordinates
• vx, vy, vz: Cartesian velocities
• rarad, decrad: Right Ascension and Declination in radians
• pmrarad, pmdecrad: Proper motion in right ascension and declination in radians
• bayer: Bayer designation
• flam: Flamsteed number
• con: Constellation abbreviation
• comp: Component identifier for multiple stars
• comp_primary: Primary component identifier for multiple stars
• base: Base temperature
• lum: Luminosity
• var: Variable star designation
• var_min, var_max: Minimum and maximum magnitudes for variable stars

Importing Libraries¶

About GridDB¶

GridDB is a reliable option to store and analyze astronomical datasets. GridDB's features, such as its high performance, scalability, and real-time analytics capabilities, make it well-suited for handling the large and complex datasets encountered in astronomy. Astronomical data often requires complex queries to extract meaningful information. GridDB supports SQL-like queries, enabling researchers and astronomers to perform various operations such as filtering, aggregation, and joining of multiple astronomical datasets and analyzing them for powerful insights. Moreover, GridDB's in-memory computing architecture, combined with its disk-based storage, enables fast data processing and low-latency access to data.

In this case, we use Python's requests package to connect to GridDB's WebAPI (GridDB Cloud) to create the data container, load data into the container, and write queries against the data.

Reading the Dataset into a Dataframe¶

We will use this dataframe to load data into the container later on.

Setting up the Authorization and Request¶

To connect to the GridDB cloud's WebAPI, the Request that is constructed should have the username and password encoded to base64 standards. To determine the base64 encoded credentials for your username and password, a tool such as https://www.base64encode.org/ can be used. Refer to the resource listed here to learn more about the different entities involved in creating a request - https://griddb.net/en/blog/using-griddb-cloud-version-1-2/. The basic authentication format in GridDB is 'username:password' where the case-sensitive username and password should be used for encoding.

Construct the base URL based on your GridDB cluster you'd like to connect to. Ensure that you replace the placeholders in the base_url below with the correct values that correspond to your GridDB instance

Creating the Data Container¶

GridDB supports two types of containers namely collections and time-series containers. While collections are general-purpose data containers that store data records with arbitrary schema, time series containers are specifically designed for handling data where each record is associated with a timestamp. In this case, Collections are ideal as they are suitable for storing diverse types of data. Refer to this official resource to learn more on GridDB containers and their characteristics.

An important pre-requisite to creating a container is to determine the data types of each data column you have. GridDB supports both primitive and complex datatypes. Among the primitive datatypes, GridDB supports Integer types, Boolean, String, Floating Point types and Time variables. Among the complex datatypes, GridDB supports Arrays, blob storage and Geospatial data storage. GridDB also has specialized datatypes that support TIMESTAMP-based data storage and retrieval optimizations and options to store Enumerated data types. Refer to this resource to learn more.

Below is a template to create a container object. For Collection containers, the rowkey can is usually set to False whereas for Timeseries containers, the rowkey has to be set to True. When we set 'rowkey' to True, we are effectively telling GridDB that the container has a primary key column. Timeseries containers require the timestamp variable to be the primary key. While Collection Containers can either have a rowkey of True or False, when the rowkey is set to False, GriDB creates a Collection Container by default.

The choice between using a row key or not depends on the requirements of your application and the access patterns you anticipate. If you need to perform frequent key-based lookups or retrieve individual records based on their unique identifiers, setting the row key to "true" would be appropriate. On the other hand, if your application primarily relies on querying and filtering records based on other field values, setting the row key to "false" is suitable.

data_obj = {
"container_name": "Container_Name",
"container_type": "Container Type",
"rowkey": False,
"columns": [
{
"name": "Name of column 1",
"type": "Datatype of column 1"
},
{ "name": "Name of column 2",
"type": "Datatype of column 2"
},
{ "name": "Name of column 3",
"type": "Datatype of column 3"
},

Rules for naming Containers and columns within a container object¶

Note that while creating column names, special characters such as '(', '/' and spaces should be avoided in column names and container names. An underscore ( _ ) can however be used. If there are such characters in your container declaration, the request fails with the error message '{"version":"v2","errorCode":145007,"errorMessage":"[145007:JC_ILLEGAL_SYMBOL_CHARACTER] Illegal character found (character=\" \"(\\u0020), component=base, containerName=\"Astronomy Dataset\", index=9)"}'. Here is a comprehensive list of error codes for GridDB - http://www.toshiba-sol.co.jp/en/pro/griddb/docs-en/v4_3/GridDB_ErrorCodes.html

A pythonic way to create a container¶

Tapping into Python's capabilities, the below code will iterate over the input_variables list and create a column dictionary for each variable. The "name" field of each column for the 'Collection' container object will be populated with the variable from the list 'input_variables'. Similarly, the 'type' field of each column for the 'Collection' container object will be populated with the variable from the list 'data_types'.

The zip() function is used to iterate over both lists simultaneously, allowing you to access the corresponding data type for each input variable. The updated data_obj will contain the dynamically generated column names based on the input variables.

Row Registration (Data Loading)¶

In GridDB, the process of loading data into a container is referred to as "Row Registration" because it involves registering or adding rows to the container. When you register a row, you are essentially inserting a new record into the container, specifying the values for each column in that row. This terminology aligns with the concept of a database table, where rows represent individual records or entities, and registering a row signifies the act of adding a new record to the table. Refer to this resource to learn more about what happens in the backend while using Row Registration. https://www.toshiba-sol.co.jp/en/pro/griddb/docs-en/v4_6/GridDB_Web_API_Reference.html#row-registration

The good thing about using Python in conjunction with GridDB is that we can use both inbuilt functions in Python and packages such as Json and Pandas to manipulate data easily and effectively. In this case, we use the 'to_json' function of the 'json' module.

The dataframe 'hygdata_v3.csv' (Refer to the section 'Reading the Dataset into a Dataframe') is used to load data into a container after a few transformations. The below lines of code convert the data from the DataFrame to JSON format (using the to_json function from the 'json' library). The JSON object is then sent in a PUT request to the GridDB WebAPI to register the rows in the specified container.

Batch Loading properties of GridDB¶

GridDB supports batch loading. If you are using the SDK, here is a useful resource - https://griddb.net/en/blog/griddb-optimization-with-multi-put-and-query/. If you use the WebAPI, then you can just use a simple for loop to load your subsets of data into your GridDB container. There are two major advantages to using batch loading to register rows into your GridDB container. One is that large datasets can be loaded seamlessly. The second is that you can effectively troubleshoot issues with your data easily.

Unveiling the Celestial Mysteries: Exploring the Captivating Astronomy Dataset¶

GridDB provides a query language called "GridDB Query Language", which is similar to SQL syntax. GQL allows you to perform various operations on GridDB databases, such as querying data, filtering results, aggregating data, and more. While GQL shares similarities with SQL, it also has its own unique features and syntax tailored specifically for GridDB's distributed architecture and data model. For Timeseries containers, GridDB provides specialized query language called TQL (TimeSeries Query Language) to support time-based aggregations, time-based selections and to set sliding windows. In this case, we stick to the SQL capabilities of GridDB as we have created a Collection Container. But if you have other use cases that are time-based, check out this TQL Reference . To refer to the SQL reference, visit this page.

GridDB supports conditional expressions like 'Case When'. Refer to this resource to learn more.

What does the distribution of the Top 20,000 stars look like?¶

Insight(s):

• When we look at the celestial co-ordinates of the Top 20,000 stars, we get the above distribution.
• The brightest star in this case has a luminosity of 400M. The next brightest seems to be having a luminosity of 250M.
• Most stars seem to be between 0 and 50M luminosity.

What are some known stars and their spectral classification and magnitude?¶

GridDB supports NULL handling and special handling. Refer to this resource to learn more.

Insight(s): In astronomy, magnitude is a logarithmic scale used to quantify the brightness of celestial objects. The lower the magnitude value, the brighter the object.

• Sirius has a magnitude of -1.44, making it one of the brightest stars in the night sky.
• In the case of 3C 273, a magnitude of 12.88 indicates that it is a relatively bright object, although not visible to the naked eye. It may be observable with telescopes or other astronomical instruments.
• Sirius, Proxima Centauri, Alpha Centauri (Rigil Kentaurus), Vega, Betelgeuse, Rigel, and Antares are noteworthy due to their brightness, proximity to Earth, or prominent positions in constellations.
• The apparent magnitudes vary significantly across the dataset, ranging from -26.7 (Sol, our Sun) to 12.88 (3C 273). This wide range represents the vast differences in luminosity between stars, with the brightest stars appearing much brighter than the dimmer ones.

What are the Top 5 Constellations by the Number of Stars?¶

GridDB provides Null Handling functions such as ISNULL, IFNULL and Coalesce among its SQL functions. Refer to this resource to know more. GridDB also supports subqueries as seen in this resource.

Insight(s):

• In terms of the number of stars, Centaurus is the largest constellation followed by Ursa Major.
• The 3rd, 4th and 5th slots are captured by Hercules, Cygnus and Hydra respectively.

Which constellations have some of the brightest stars?¶

Insight(s): Below is a summary of the constellations with the brightest stars:

• The star Sol("Solaris") referring to the Sun tops the list. It is not considered a part of any constellation in the traditional sense. The Sun, being the star at the center of our solar system, does not fall within the framework of the 88 constellations and hence does not have a constellation name.
• The next brightest star is Sirius that belongs to the Constellation CMa('Canis Major').
• This is followed by Canopus that belongs to the Constellation Car('Carina').

Visualizing the 'Orion' Constellation¶

Insight(s): The above plot shows the coordinates of all stars in the Orion Constellation.

What are the spectral types for known stars across all constellations?¶

Insight(s):

• The Hercules constellation has more than 500 stars with the K0 spectral type.
• K0 stars have a temperature range of approximately 4,500 to 5,000 Kelvin and generally appear orange in color.
• A lot of other constellations like Ursa Major, Camelopardalis, Virgo, etc. and have between 201 and 500 K0 stars.
G-type stars, also known as yellow dwarf stars, include stars like our Sun (classified as G2V). The average temperature of G-type stars is around 5,000 to 6,000 Kelvin (K). Ursa Major, Virgo, Boötes and Cetus have between 201 and 500 G0 stars.
• It's important to note that Ursa Major, Virgo, Boötes and Cetus also have an enormous amount of G5 stars.
• Hercules, Draco, Leo and Pisces have G5 stars too.
• G0 stars are hotter than G5 stars and have temperatures closer to the upper end of the G-type range. G0 stars have temperatures around 5,900 to 6,000 Kelvin, while G5 stars have temperatures around 5,500 to 5,600 Kelvin.

What does the spectral distribution of Stars look like across each constellation?¶

Here, we use GridDB's Conditional Expressions to create categories of spectral types and to create bins of the number of stars in each spectral type and constellation.

Insight(s):

• C-type, M-N types and K-type stars seem to be widely distributed across all constellations.
• Aquila(Aql),Octans(Oct), Crux(Cru) and Lepus(Lep) are some constellations that have G-type stars also known as G-dwarfs or yellow dwarf stars.
• O-type stars are less common. Dorado(Dor), Leo Minor(Lmi), Coma Berenices(Com), Chamaeleon(Cha) and a couple more constellations have O-type stars.

Concluding Remarks¶

In conclusion, the analysis conducted on the Astronomy Nexus data using GridDB has provided valuable insights into the complex world of astronomy. The utilization of GridDB's powerful features, such as subqueries and case statements, has greatly enhanced the effectiveness and efficiency of the analysis process.

Through the implementation of subqueries, we were able to extract specific subsets of data, allowing for targeted analysis on various astronomical phenomena. This enabled us to explore specific aspects of the data and uncover correlations and patterns that would have otherwise been challenging to identify.

The utilization of case statements within GridDB offered a flexible and intuitive way to handle complex conditions and derive meaningful information from the data. By applying case when statements and Null Handling, we were able to categorize and classify astronomical objects based on their specific characteristics, facilitating a more comprehensive understanding of the data set.

The Astronomy Nexus data presented unique challenges due to its complexity and vastness. However, GridDB proved to be an invaluable tool in handling and processing such intricate data. Its ability to efficiently store, retrieve, and analyze massive datasets while providing high availability and scalability allowed for seamless exploration and manipulation of the Astronomy Nexus data.

In summary, the combination of GridDB's advanced features, including subqueries and case statements, with the rich and intricate Astronomy Nexus data has demonstrated the effectiveness and utility of GridDB in the field of astronomy. By leveraging GridDB's capabilities, astronomers and researchers can unlock deeper insights and make significant advancements in our understanding of the universe.