This repository contains notebooks based on the Virtual Observatory example Python scripts from https://github.com/hendhd/hyantis/pysrc/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 

355 lines
14 KiB

{
"cells": [
{
"cell_type": "markdown",
"id": "reflected-novel",
"metadata": {},
"source": [
"# Find associated EVN data for your favourite target or sample"
]
},
{
"cell_type": "markdown",
"id": "closing-differential",
"metadata": {},
"source": [
"This notebook is developed to demonstrate how you can use PyVO to perform a basic cross-match between a VO-catalogue and a VO-service. It demonstrates the following steps:\n",
"* how to access a VO TAP service\n",
"* search for a Vizier catalogue given a description keyword (following the [instructions from Katharina Lutz](https://www.astrobetter.com/blog/2020/06/29/the-cds-and-python-iii-vizier-xmatch-20k-catalogues-and-tables-at-your-fingertips/))\n",
"* explain the basic ADQL query for a cross match\n",
"* do a cross-match based on RA and DEC\n",
"* broadcast the results to topcat\n",
"* OPTIONAL: search for data in additional VO services\n",
"\n",
"As a test case, I have chosen the LOFAR DDRG sample published by Mahatma et al (2018). This sample is already in Vizier, so no need to download it and get it into a file, the whole idea of the VO is that you can access the tables remotely. The table identifier is J/A+A/622/A13. "
]
},
{
"cell_type": "markdown",
"id": "first-africa",
"metadata": {},
"source": [
"#### 1. Get the necessary imports\n",
"The imports for the notebook include the PyVO module. This is not standard in the anaconda distribution. You can [follow these instrunctions](https://anaconda.org/conda-forge/pyvo)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "confident-accommodation",
"metadata": {},
"outputs": [],
"source": [
"# Usual imports, this takes a bit of time\n",
"import pyvo\n",
"from astropy.table import Table\n",
"from astropy.coordinates import SkyCoord"
]
},
{
"cell_type": "markdown",
"id": "covered-modification",
"metadata": {},
"source": [
"#### 2. Identify the TAP service\n",
"The first step to find a VO table in Vizier, is to tell PyVO where the service can be found. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fuzzy-liver",
"metadata": {},
"outputs": [],
"source": [
"# Identify the service at Vizier\n",
"tap_vizier= pyvo.dal.TAPService('http://tapvizier.u-strasbg.fr/TAPVizieR/tap/')"
]
},
{
"cell_type": "markdown",
"id": "recorded-intro",
"metadata": {},
"source": [
"#### 3. Search for a table and obtain its content\n",
"All VO tables come with a description, which can be very helpful, but not all authors are aware how this is used in the VO environment. E.g. the example below searches for a table of double-double radio galaxies observed with LOFAR. The table description does not contain either of the keywords you would expect, but luckily the name of the first author is added. \n",
"\n",
"The search is done with an ADQL query, the text between brackets, and explicitly written to a new table called `my_search`. The keywords `tables`, `table_name` and `description` are standard keywords for querying Vizier. Note the specific way a text is quoted. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "processed-british",
"metadata": {},
"outputs": [],
"source": [
"# Search for tables in Vizier containing the word '3CR' in the description\n",
"# How would you select a table by its name?\n",
"my_search = tap_vizier.search(\"SELECT * FROM tables \" +\n",
" \"WHERE description LIKE '%3CR%'\").to_table()\n",
"\n",
"# Print the list of tables that match the criteria, only the name and description\n",
"# For the 3CR search this is a decent list!\n",
"my_search['table_name','description']"
]
},
{
"cell_type": "markdown",
"id": "joint-family",
"metadata": {},
"source": [
"From the list of tables found in Vizier, select the `table_name` of the table you are interested in. This goes into the ADQL query below to generate a new table. \n",
"NOTE: the search above does not give the same results as the search on the Vizier website, because it only searches in the description. Need to cross-match name and description to get everything."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "military-issue",
"metadata": {},
"outputs": [],
"source": [
"# Identify the table needed and select all records\n",
"# Note that the name of the table requires the 'list' part \n",
"my_table = tap_vizier.search(\"SELECT * FROM \\\"J/ApJS/164/307/catalog\\\" \").to_table()\n",
"\n",
"#my_table = tap_vizier.search(\"SELECT Name,_RA,_DE FROM \\\"J/MNRAS/440/269/3CRR\\\" \").to_table()\n",
"#my_table = tap_vizier.search(\"SELECT 3CR,RA1950,DE1950 FROM \\\"J/PASP/97/932/3cr\\\" \").to_table()\n",
"#my_table = tap_vizier.search(\"SELECT RAJ2000,DEJ2000,NAME FROM \\\"J/MNRAS/404/1719/tablea2\\\" \").to_table()\n",
"\n",
"#my_table = tap_vizier.search(\"SELECT 3C as Name,RAJ2000,DEJ2000 FROM \\\"J/ApJS/164/307/catalog\\\" \").to_table()\n",
"\n",
"# Figure out the full length of the table\n",
"#print(\"Number of table entries: \" + str(len(my_table.columns['RAJ2000'])))\n",
"\n",
"# Print the first 10 records of the table\n",
"my_table[0:10]"
]
},
{
"cell_type": "markdown",
"id": "integral-benchmark",
"metadata": {},
"source": [
"For older tables the coordinates are in B1950, and not J2000. The JOIN command further down should take care of this by using the ICRS reference frame. Need to check this, I don't expect a basic ADQL query to do anything like this.\n",
"Otherwise, the astropy coordinates package can be used to ensure all coordinates are in the ICRS frame."
]
},
{
"cell_type": "markdown",
"id": "diverse-suspension",
"metadata": {},
"source": [
"#### 4. Cross match with another VO service\n",
"For each element in `my_table` we want to search the EVN archive. Conveniently, the EVN archive also runs a TAP service that can be used for this. Define the EVN TAP service, this is the same as step 2 shown above."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "functional-gossip",
"metadata": {},
"outputs": [],
"source": [
"# Now point to EVN TAP service\n",
"tap_evn = pyvo.dal.TAPService('http://evn-vo.jive.eu/tap')"
]
},
{
"cell_type": "markdown",
"id": "narrative-belize",
"metadata": {},
"source": [
"#### 5. Define and perform the ADQL query for a cross match\n",
"In this step we define the ADQL query, which defines how to cross-match the two tables. The search can be done within a specific region, or even within the EVN field-of-view. One way to handle this, is using an ADQL `JOIN` command. \n",
"\n",
"The query works as follows:\n",
"* `SELECT` is the default ADQL command to select information from tables\n",
"* `*` indidates that all columns in the table are to be selected, and the resulting table will be called `db` in the query\n",
"* `FROM` points to the default table `ivoa.obscore`, which is present in all VO services\n",
"* `JOIN` takes the elements that are present in both tables \n",
"* `TAP_UPLOAD` ensures that the table identified above is included in the query, it will be called `mine`\n",
"* `ON 1 = CONTAINS ()` is a boolean statement that is true if the condition between brackets is met, in this case it searches for a point location in a circle of a given radius\n",
"* `POINT` defines a location in ICRS coordinates, it uses the exact keywords in `my_table`\n",
"* `CIRCLE` defines a circular region around the ICRS coordinates of a given radius (in this case the FoV) \n",
"\n",
"Note that if this doesn't work for you, most likely the RA and DEC keywords in your table are different. Make sure to check above in the table output what the exact keywords are."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "average-tanzania",
"metadata": {},
"outputs": [],
"source": [
"QUERY=\"\"\"\n",
"SELECT\n",
" *\n",
" FROM ivoa.obscore AS db\n",
" JOIN TAP_UPLOAD.my_table AS mine\n",
" ON 1=CONTAINS(POINT('ICRS',mine.RAJ2000, mine.DEJ2000),\n",
" CIRCLE('ICRS',db.s_ra, db.s_dec, 0.1))\n",
"\"\"\"\n",
"# Note that using the FOV in the circle definition does not work well for EVN, as the FOV is extremely small. \n",
"# Therefore it is better to set it to a numeric value, in this cae 0.01 degrees, which may be too large."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "unknown-scope",
"metadata": {},
"outputs": [],
"source": [
"# Perform an ADQL query on the service, using RA and DEC from the 3CRR table, and make the results a new VO table\n",
"\n",
"evn_search = tap_evn.run_sync(QUERY, uploads={\"my_table\":my_table})"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "private-board",
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"# Look at the resulting table (empty in case of this example)\n",
"results_table = evn_search.to_table()\n",
"print(\"Number of table entires: \" + str(len(results_table.columns['Name'])))\n",
"\n",
"# Can I get these links as active, clickable links?\n",
"\n",
"results_table['Name','target_name','obs_id','access_url'][0:10]\n",
"#results_table\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "informative-syria",
"metadata": {},
"source": [
"#### 6. Get the results in topcat\n",
"If you are familiar with topcat, this is a good point to launch the service. The resulting table can be broadcast to topcat via `SAMP`.\n",
"\n",
"In topcat you can also look at the information a TAP service provides: select VO --> TAP Query, and add the TAP service URL on the bottom line, then click Use Service."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "functional-information",
"metadata": {},
"outputs": [],
"source": [
"# If you launch TOPCAT before running the notebook, uncomment the last line in this cell\n",
"# and the results will be listed there.\n",
"# WARNINGS given as output are usually benign\n",
"\n",
"evn_search.broadcast_samp(\"topcat\")"
]
},
{
"cell_type": "markdown",
"id": "simplified-homeless",
"metadata": {},
"source": [
"#### 7. Get your data\n",
"At the moment the easiest way to actually get the visibility data is to select the observation ID and go to the [EVN Archive website](http://archive.jive.nl). Very very soon there will be an alternative route using the [JIVE Science Platform](https://jupyterhub.jive.eu), which runs a JupyterHub server where you can process the data locally on JIVE machines using a Jupyter notebook. At the moment this only works for continuum observations."
]
},
{
"cell_type": "markdown",
"id": "retained-campaign",
"metadata": {},
"source": [
"#### 8. Next steps and summary\n",
"\n",
"If the resulting table is empty, but you do expect results, there are some options to try:\n",
"- widen the search area (note that trying to increase the radius to >0.1 degree causes a fail in some cases)\n",
"- generate the circle for the database with the largest FoV\n",
"\n",
"If there's no EVN data for your sample: go write a proposal!"
]
},
{
"cell_type": "markdown",
"id": "residential-ticket",
"metadata": {},
"source": [
"Alternatively, a query like this can be expanded to search for *any* data for these objects in all VO services. This is very time consuming and doing it from a notebook means you will have to wait it out."
]
},
{
"cell_type": "markdown",
"id": "discrete-gender",
"metadata": {},
"source": [
"#### Additional example: query all VO services\n",
"Note that this step is extremely time consuming and cannot easily be terminated. It is included as an example. The script is based on example 4 in the hyantis/pysrc examples from Hendrik Heinl ([github link](https://github.com/hendhd/hyantis/tree/main/pysrc))."
]
},
{
"cell_type": "raw",
"id": "prime-alberta",
"metadata": {
"scrolled": false
},
"source": [
"# TO RUN THIS CELL CONVERT TO CODE\n",
"# Copy and pasted from hyantis' example 4. This can take a while to run.\n",
"# Note that the query is inverted wrt the example, assuming all obscore tables have a FOV value\n",
"# This seems to work OK, the WARNINGs can again be safely ignored.\n",
"\n",
"# Make a list of all the VO services to search\n",
"obsvc=pyvo.registry.search(datamodel=\"obscore\")\n",
"\n",
"# For each service, check if there is data:\n",
"for row in obsvc:\n",
" \n",
" try:\n",
" print (\"Querying {url}\".format(url=row.access_url))\n",
" service=pyvo.dal.TAPService(row.access_url)\n",
" my_result=service.run_async(query=QUERY, uploads={\"my_table\":my_table})\n",
" my_result.broadcast_samp(\"topcat\")\n",
" \n",
" except Exception as msg:\n",
" # some service is broken; you *should* complain, but then let's be lazy here\n",
" print(\" Broken: {} ({})\\n\".format(row.access_url, msg))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "parliamentary-marble",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}