This repository contains notebooks based on the Virtual Observatory example Python scripts from https://github.com/hendhd/hyantis/pysrc/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 

322 lines
12 KiB

{
"cells": [
{
"cell_type": "markdown",
"id": "reflected-novel",
"metadata": {},
"source": [
"# Find associated EVN data for your favourite target or sample"
]
},
{
"cell_type": "markdown",
"id": "closing-differential",
"metadata": {},
"source": [
"This notebook is developed to demonstrate how you can use PyVO to perform a basic cross-match between a VO-catalogue and a VO-service. It demonstrates the following steps:\n",
"* how to access a VO TAP service\n",
"* search for a Vizier catalogue given a description keyword (following the [instructions from Katharina Lutz](https://www.astrobetter.com/blog/2020/06/29/the-cds-and-python-iii-vizier-xmatch-20k-catalogues-and-tables-at-your-fingertips/))\n",
"* explain the basic ADQL query for a cross match\n",
"* do a cross-match based on RA and DEC\n",
"* broadcast the results to topcat\n",
"* OPTIONAL: search for data in additional VO services\n",
"\n",
"As a test case, I have chosen the LOFAR DDRG sample published by Mahatma et al (2018). This sample is already in Vizier, so no need to download it and get it into a file, the whole idea of the VO is that you can access the tables remotely. The table identifier is J/A+A/622/A13. "
]
},
{
"cell_type": "markdown",
"id": "first-africa",
"metadata": {},
"source": [
"#### 1. Get the necessary imports\n",
"The imports for the notebook include the PyVO module. This is not standard in the anaconda distribution. You can [follow these instrunctions](https://anaconda.org/conda-forge/pyvo)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "confident-accommodation",
"metadata": {},
"outputs": [],
"source": [
"# Usual imports, this takes a bit of time\n",
"import pyvo\n",
"from astropy.table import Table"
]
},
{
"cell_type": "markdown",
"id": "covered-modification",
"metadata": {},
"source": [
"#### 2. Identify the TAP service\n",
"The first step to find a VO table in Vizier, is to tell PyVO where the service can be found. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fuzzy-liver",
"metadata": {},
"outputs": [],
"source": [
"# Identify the service at Vizier\n",
"tap_vizier= pyvo.dal.TAPService('http://tapvizier.u-strasbg.fr/TAPVizieR/tap/')"
]
},
{
"cell_type": "markdown",
"id": "recorded-intro",
"metadata": {},
"source": [
"#### 3. Search for a table and obtain its content\n",
"All VO tables come with a description, which can be very helpful, but not all authors are aware how this is used in the VO environment. E.g. the example below searches for a table of double-double radio galaxies observed with LOFAR. The table description does not contain either of the keywords you would expect, but luckily the name of the first author is added. \n",
"\n",
"The search is done with an ADQL query, the text between brackets, and explicitly written to a new table called `ddrg_search`. The keywords `tables`, `table_name` and `description` are standard keywords for querying Vizier. Note the specific way a text is quoted. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "processed-british",
"metadata": {},
"outputs": [],
"source": [
"# Search for tables in Vizier containing the word 'Mahatma' (first author)\n",
"my_search = tap_vizier.search(\"SELECT * FROM tables \" +\n",
" \"WHERE description LIKE '%Mahatma%'\").to_table()\n",
"\n",
"# Print the list of tables that match the criteria, only the name and description\n",
"my_search['table_name','description']"
]
},
{
"cell_type": "markdown",
"id": "joint-family",
"metadata": {},
"source": [
"From the list of tables found in Vizier, select the `table_name` of the table you are interested in. This goes into the ADQL query below to generate a new table."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "military-issue",
"metadata": {},
"outputs": [],
"source": [
"# Identify the table needed and select all records\n",
"# Note that the name of the table requires the 'list' part \n",
"my_table = tap_vizier.search(\"SELECT * FROM \\\"J/A+A/622/A13/list\\\" \").to_table()\n",
"\n",
"# Print the first 10 records of the table\n",
"my_table[0:10]"
]
},
{
"cell_type": "markdown",
"id": "diverse-suspension",
"metadata": {},
"source": [
"#### 4. Cross match with another VO service\n",
"For each element in `my_table` we want to search the EVN archive. Conveniently, the EVN archive also runs a TAP service that can be used for this. Define the EVN TAP service, this is the same as step 2 shown above. \n",
"\n",
"Side note: to avoid lengthy searches, it is possible to first look at the sky overlap between two surveys using the MOCS or HIPS information, which encodes where a survey has data."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "functional-gossip",
"metadata": {},
"outputs": [],
"source": [
"# Now point to EVN TAP service\n",
"tap_evn = pyvo.dal.TAPService('http://evn-vo.jive.eu/tap')"
]
},
{
"cell_type": "markdown",
"id": "narrative-belize",
"metadata": {},
"source": [
"#### 5. Define and perform the ADQL query for a cross match\n",
"Then we define the ADQL query. The search can be done within a specific region, or even within the EVN field-of-view. One way to handle this, is using an ADQL `JOIN` command. \n",
"\n",
"The query works as follows:\n",
"* `SELECT` is the default ADQL command to select information from tables\n",
"* `*` indidates that all columns in the table are to be selected, and the resulting table will be called `db` in the query\n",
"* `FROM` points to the default table `ivoa.obscore`, which is present in all VO services\n",
"* `JOIN` takes the elements that are present in both tables \n",
"* `TAP_UPLOAD` ensures that the table identified above is included in the query, it will be called `mine`\n",
"* `ON 1 = CONTAINS ()` is a boolean statement that is true if the condition between brackets is met, in this case it searches for a point location in a circle of a given radius\n",
"* `POINT` defines a location in ICRS coordinates, it uses the exact keywords in `my_table`\n",
"* `CIRCLE` defines a circular region around the ICRS coordinates of a given radius (in this case the FoV) \n",
"\n",
"Note that if this doesn't work for you, most likely the RA and DEC keywords in your table are different. Make sure to check above in the table output what the exact keywords are."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "average-tanzania",
"metadata": {},
"outputs": [],
"source": [
"QUERY=\"\"\"\n",
"SELECT\n",
" *\n",
" FROM ivoa.obscore AS db\n",
" JOIN TAP_UPLOAD.my_table AS mine\n",
" ON 1=CONTAINS(POINT('ICRS',mine.RAJ2000, mine.DEJ2000),\n",
" CIRCLE('ICRS',db.s_ra, db.s_dec, db.s_fov))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "unknown-scope",
"metadata": {},
"outputs": [],
"source": [
"# Perform an ADQL query on the service, using RA and DEC from the DDRG table.\n",
"\n",
"evn_search = tap_evn.run_sync(QUERY, uploads={\"my_table\":my_table})"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "private-board",
"metadata": {},
"outputs": [],
"source": [
"# Look at the resulting table (empty in case of this example)\n",
"evn_search"
]
},
{
"cell_type": "markdown",
"id": "informative-syria",
"metadata": {},
"source": [
"#### 6. Get the results in topcat\n",
"If you are familiar with topcat, this is a good point to launch the service. The resulting table can be broadcast to topcat via `SAMP`.\n",
"\n",
"In topcat you can also look at the information a TAP service provides: select VO --> TAP Query, and add the TAP service URL on the bottom line, then click Use Service."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "functional-information",
"metadata": {},
"outputs": [],
"source": [
"# If you launch TOPCAT before running the notebook, the results will be listed there\n",
"# WARNINGS given as output are usually benign\n",
"evn_search.broadcast_samp(\"topcat\")"
]
},
{
"cell_type": "markdown",
"id": "simplified-homeless",
"metadata": {},
"source": [
"In this example there is no cross-match between the LOFAR and EVN observations. You can try to find another table (or tables) to cross-match, or you can first inspect the sky-coverage of the surveys using the Aladin MOCs visualization. "
]
},
{
"cell_type": "markdown",
"id": "retained-campaign",
"metadata": {},
"source": [
"#### Next steps and summary\n",
"\n",
"If the resulting table is empty, but you do expect results, there are some options to try:\n",
"- widen the search area (note that trying to increase the radius to >0.1 degree causes a fail)\n",
"- generate the circle for the database with the largest FoV\n",
"\n",
"If there's no EVN data for your sample: go write a proposal!"
]
},
{
"cell_type": "markdown",
"id": "residential-ticket",
"metadata": {},
"source": [
"Alternatively, a query like this can be expanded to search for *any* data for these objects in all VO services. This is very time consuming and doing it from a notebook means you will have to wait it out."
]
},
{
"cell_type": "markdown",
"id": "discrete-gender",
"metadata": {},
"source": [
"#### Additional example: query all VO services\n",
"Note that this step is extremely time consuming and cannot easily be terminated. It is included as an example. The script is based on example 4 in the hyantis/pysrc examples from Hendrik Heinl ([github link](https://github.com/hendhd/hyantis/tree/main/pysrc))."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "mysterious-census",
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"# Copy and pasted from 'example 4'. This can take a while to run.\n",
"# Note that the query is inverted wrt the example, assuming all obscore tables have a FOV value\n",
"# This seems to work OK, the WARNINGs can again be safely ignored.\n",
"\n",
"# Make a list of all the VO services to search\n",
"obsvc=pyvo.registry.search(datamodel=\"obscore\")\n",
"\n",
"# For each service, check if there is data:\n",
"for row in obsvc:\n",
" \n",
" try:\n",
" print (\"Querying {url}\".format(url=row.access_url))\n",
" service=pyvo.dal.TAPService(row.access_url)\n",
" my_result=service.run_async(query=QUERY, uploads={\"my_table\":my_table})\n",
" my_result.broadcast_samp(\"topcat\")\n",
" \n",
" except Exception as msg:\n",
" # some service is broken; you *should* complain, but then let's be lazy here\n",
" print(\" Broken: {} ({})\\n\".format(row.access_url, msg))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "parliamentary-marble",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}