Browse Source

New edu notebook and updates

master
Ilse van Bemmel 7 months ago
parent
commit
329be38efd
  1. 320
      EVN PyVO edu.ipynb
  2. 574
      EVN PyVO interface.ipynb

320
EVN PyVO edu.ipynb

@ -0,0 +1,320 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "reflected-novel",
"metadata": {},
"source": [
"# Find associated EVN data for your favourite target or sample"
]
},
{
"cell_type": "markdown",
"id": "closing-differential",
"metadata": {},
"source": [
"This notebook is developed to demonstrate how you can use PyVO to perform a basic cross-match between a VO-catalogue and a VO-service. It demonstrates the following steps:\n",
"* how to access a VO TAP service\n",
"* search for a Vizier catalogue given a description keyword (following the [instructions from Katharina Lutz](https://www.astrobetter.com/blog/2020/06/29/the-cds-and-python-iii-vizier-xmatch-20k-catalogues-and-tables-at-your-fingertips/))\n",
"* explain the basic ADQL query for a cross match\n",
"* do a cross-match based on RA and DEC\n",
"* broadcast the results to topcat\n",
"* OPTIONAL: search for data in additional VO services\n",
"\n",
"As a test case, I have chosen the LOFAR DDRG sample published by Mahatma et al (2018). This sample is already in Vizier, so no need to download it and get it into a file, the whole idea of the VO is that you can access the tables remotely. The table identifier is J/A+A/622/A13. "
]
},
{
"cell_type": "markdown",
"id": "first-africa",
"metadata": {},
"source": [
"#### 1. Get the necessary imports\n",
"The imports for the notebook include the PyVO module. This is not standard in the anaconda distribution. You can [follow these instrunctions](https://anaconda.org/conda-forge/pyvo)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "confident-accommodation",
"metadata": {},
"outputs": [],
"source": [
"# Usual imports, this takes a bit of time\n",
"import pyvo\n",
"from astropy.table import Table"
]
},
{
"cell_type": "markdown",
"id": "covered-modification",
"metadata": {},
"source": [
"#### 2. Identify the TAP service\n",
"The first step to find a VO table in Vizier, is to tell PyVO where the service can be found. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fuzzy-liver",
"metadata": {},
"outputs": [],
"source": [
"# Identify the service at Vizier\n",
"tap_vizier= pyvo.dal.TAPService('http://tapvizier.u-strasbg.fr/TAPVizieR/tap/')"
]
},
{
"cell_type": "markdown",
"id": "recorded-intro",
"metadata": {},
"source": [
"#### 3. Search for a table and obtain its content\n",
"All VO tables come with a description, which can be very helpful, but not all authors are aware how this is used in the VO environment. E.g. the example below searches for a table of double-double radio galaxies observed with LOFAR. The table description does not contain either of the keywords you would expect, but luckily the name of the first author is added. \n",
"\n",
"The search is done with an ADQL query, the text between brackets, and explicitly written to a new table called `ddrg_search`. The keywords `tables`, `table_name` and `description` are standard keywords for querying Vizier. Note the specific way a text is quoted. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "processed-british",
"metadata": {},
"outputs": [],
"source": [
"# Search for tables in Vizier containing the word 'Mahatma' (first author)\n",
"my_search = tap_vizier.search(\"SELECT * FROM tables \" +\n",
" \"WHERE description LIKE '%Mahatma%'\").to_table()\n",
"\n",
"# Print the list of tables that match the criteria, only the name and description\n",
"my_search['table_name','description']"
]
},
{
"cell_type": "markdown",
"id": "joint-family",
"metadata": {},
"source": [
"From the list of tables found in Vizier, select the `table_name` of the table you are interested in. This goes into the ADQL query below to generate a new table."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "military-issue",
"metadata": {},
"outputs": [],
"source": [
"# Identify the table needed and select all records\n",
"# Note that the name of the table requires the 'list' part \n",
"my_table = tap_vizier.search(\"SELECT * FROM \\\"J/A+A/622/A13/list\\\" \").to_table()\n",
"\n",
"# Print the first 10 records of the table\n",
"my_table[0:10]"
]
},
{
"cell_type": "markdown",
"id": "diverse-suspension",
"metadata": {},
"source": [
"#### 4. Cross match with another VO service\n",
"For each element in `my_table` we want to search the EVN archive. Conveniently, the EVN archive also runs a TAP service that can be used for this. Define the EVN TAP service, this is the same as step 2 shown above."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "functional-gossip",
"metadata": {},
"outputs": [],
"source": [
"# Now point to EVN TAP service\n",
"tap_evn = pyvo.dal.TAPService('http://evn-vo.jive.eu/tap')"
]
},
{
"cell_type": "markdown",
"id": "narrative-belize",
"metadata": {},
"source": [
"#### 5. Define and perform the ADQL query for a cross match\n",
"Then we define the ADQL query. The search can be done within a specific region, or even within the EVN field-of-view. One way to handle this, is using an ADQL `JOIN` command. \n",
"\n",
"The query works as follows:\n",
"* `SELECT` is the default ADQL command to select information from tables\n",
"* `*` indidates that all columns in the table are to be selected, and the resulting table will be called `db` in the query\n",
"* `FROM` points to the default table `ivoa.obscore`, which is present in all VO services\n",
"* `JOIN` takes the elements that are present in both tables \n",
"* `TAP_UPLOAD` ensures that the table identified above is included in the query, it will be called `mine`\n",
"* `ON 1 = CONTAINS ()` is a boolean statement that is true if the condition between brackets is met, in this case it searches for a point location in a circle of a given radius\n",
"* `POINT` defines a location in ICRS coordinates, it uses the exact keywords in `my_table`\n",
"* `CIRCLE` defines a circular region around the ICRS coordinates of a given radius (in this case the FoV) \n",
"\n",
"Note that if this doesn't work for you, most likely the RA and DEC keywords in your table are different. Make sure to check above in the table output what the exact keywords are."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "average-tanzania",
"metadata": {},
"outputs": [],
"source": [
"QUERY=\"\"\"\n",
"SELECT\n",
" *\n",
" FROM ivoa.obscore AS db\n",
" JOIN TAP_UPLOAD.my_table AS mine\n",
" ON 1=CONTAINS(POINT('ICRS',mine.RAJ2000, mine.DEJ2000),\n",
" CIRCLE('ICRS',db.s_ra, db.s_dec, db.s_fov))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "unknown-scope",
"metadata": {},
"outputs": [],
"source": [
"# Perform an ADQL query on the service, using RA and DEC from the DDRG table.\n",
"\n",
"evn_search = tap_evn.run_sync(QUERY, uploads={\"my_table\":my_table})"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "private-board",
"metadata": {},
"outputs": [],
"source": [
"# Look at the resulting table (empty in case of this example)\n",
"evn_search"
]
},
{
"cell_type": "markdown",
"id": "informative-syria",
"metadata": {},
"source": [
"#### 6. Get the results in topcat\n",
"If you are familiar with topcat, this is a good point to launch the service. The resulting table can be broadcast to topcat via `SAMP`.\n",
"\n",
"In topcat you can also look at the information a TAP service provides: select VO --> TAP Query, and add the TAP service URL on the bottom line, then click Use Service."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "functional-information",
"metadata": {},
"outputs": [],
"source": [
"# If you launch TOPCAT before running the notebook, the results will be listed there\n",
"# WARNINGS given as output are usually benign\n",
"evn_search.broadcast_samp(\"topcat\")"
]
},
{
"cell_type": "markdown",
"id": "simplified-homeless",
"metadata": {},
"source": [
"In this example there is no cross-match between the LOFAR and EVN observations. You can try to find another table (or tables) to cross-match, or you can first inspect the sky-coverage of the surveys using the Aladin MOCs visualization. "
]
},
{
"cell_type": "markdown",
"id": "retained-campaign",
"metadata": {},
"source": [
"#### Next steps and summary\n",
"\n",
"If the resulting table is empty, but you do expect results, there are some options to try:\n",
"- widen the search area (note that trying to increase the radius to >0.1 degree causes a fail)\n",
"- generate the circle for the database with the largest FoV\n",
"\n",
"If there's no EVN data for your sample: go write a proposal!"
]
},
{
"cell_type": "markdown",
"id": "residential-ticket",
"metadata": {},
"source": [
"Alternatively, a query like this can be expanded to search for *any* data for these objects in all VO services. This is very time consuming and doing it from a notebook means you will have to wait it out."
]
},
{
"cell_type": "markdown",
"id": "discrete-gender",
"metadata": {},
"source": [
"#### Additional example: query all VO services\n",
"Note that this step is extremely time consuming and cannot easily be terminated. It is included as an example. The script is based on example 4 in the hyantis/pysrc examples from Hendrik Heinl ([github link](https://github.com/hendhd/hyantis/tree/main/pysrc))."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "mysterious-census",
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"# Copy and pasted from 'example 4'. This can take a while to run.\n",
"# Note that the query is inverted wrt the example, assuming all obscore tables have a FOV value\n",
"# This seems to work OK, the WARNINGs can again be safely ignored.\n",
"\n",
"# Make a list of all the VO services to search\n",
"obsvc=pyvo.registry.search(datamodel=\"obscore\")\n",
"\n",
"# For each service, check if there is data:\n",
"for row in obsvc:\n",
" \n",
" try:\n",
" print (\"Querying {url}\".format(url=row.access_url))\n",
" service=pyvo.dal.TAPService(row.access_url)\n",
" my_result=service.run_async(query=QUERY, uploads={\"my_table\":my_table})\n",
" my_result.broadcast_samp(\"topcat\")\n",
" \n",
" except Exception as msg:\n",
" # some service is broken; you *should* complain, but then let's be lazy here\n",
" print(\" Broken: {} ({})\\n\".format(row.access_url, msg))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "parliamentary-marble",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

574
EVN PyVO interface.ipynb

@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "assisted-humor",
"id": "reflected-novel",
"metadata": {},
"source": [
"# Get the LOFAR DDRG sample and find EVN data"
@ -11,7 +11,7 @@
{
"attachments": {},
"cell_type": "markdown",
"id": "wired-payment",
"id": "closing-differential",
"metadata": {},
"source": [
"The LOFAR DDRG sample is already in Vizier, so no need to download it and get it into a file, the whole idea of the VO is that you can access the tables remotely. \n",
@ -22,7 +22,7 @@
{
"attachments": {},
"cell_type": "markdown",
"id": "statewide-england",
"id": "sufficient-integral",
"metadata": {},
"source": [
"Following the [instructions from Katharina Lutz](https://www.astrobetter.com/blog/2020/06/29/the-cds-and-python-iii-vizier-xmatch-20k-catalogues-and-tables-at-your-fingertips/), first try to read the DDRG table from Vizier.\n"
@ -31,7 +31,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "forty-suicide",
"id": "confident-accommodation",
"metadata": {},
"outputs": [],
"source": [
@ -43,7 +43,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "heavy-breath",
"id": "processed-british",
"metadata": {},
"outputs": [],
"source": [
@ -62,7 +62,7 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "historic-screening",
"id": "military-issue",
"metadata": {},
"outputs": [
{
@ -140,16 +140,16 @@
},
{
"cell_type": "markdown",
"id": "traditional-retrieval",
"id": "narrative-belize",
"metadata": {},
"source": [
"For each element in the table DDRG I want to search the EVN TAP service. This means taking the RA and DEC for each object, and giving that to the EVN service."
"For each element in the table DDRG I want to search the EVN TAP service. For every object in the DDRG table I take the RA and DEC, and check the EVN archive for data within 0.1 degree of that position. This is done with and SQL JOIN command. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "celtic-dealing",
"execution_count": 40,
"id": "average-tanzania",
"metadata": {},
"outputs": [],
"source": [
@ -158,36 +158,23 @@
" *\n",
" FROM ivoa.obscore AS db\n",
" JOIN TAP_UPLOAD.ddrg AS mine\n",
" ON 1=CONTAINS(POINT('ICRS',db.s_ra, db.s_dec),\n",
" CIRCLE('ICRS',mine.RAJ2000, mine.DECJ2000, 0.1))\n",
" ON 1=CONTAINS(POINT('ICRS',mine.RAJ2000, mine.DEJ2000),\n",
" CIRCLE('ICRS',db.s_ra, db.s_dec, db.s_fov))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "extended-teens",
"execution_count": 41,
"id": "functional-information",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:254: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:303: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"ename": "DALQueryError",
"evalue": "Query Error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mDALQueryError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-5-50512450c2b4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0;31m# Perform an ADQL query on the service, using RA and DEC from the DDRG table.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0mevn_search\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtap_evn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mQUERY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muploads\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m\"ddrg\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mddrg\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0;31m# If you launch TOPCAT before running the notebook, the results will be listed there\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/opt/anaconda3/lib/python3.8/site-packages/pyvo/dal/tap.py\u001b[0m in \u001b[0;36mrun_async\u001b[0;34m(self, query, language, maxrec, uploads, **keywords)\u001b[0m\n\u001b[1;32m 291\u001b[0m self.baseurl, query, language, maxrec, uploads, self._session, **keywords)\n\u001b[1;32m 292\u001b[0m \u001b[0mjob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 293\u001b[0;31m \u001b[0mjob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_if_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 294\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfetch_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 295\u001b[0m \u001b[0mjob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdelete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/opt/anaconda3/lib/python3.8/site-packages/pyvo/dal/tap.py\u001b[0m in \u001b[0;36mraise_if_error\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 822\u001b[0m \"\"\"\n\u001b[1;32m 823\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mphase\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\"ERROR\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"ABORTED\"\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 824\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mDALQueryError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Query Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mphase\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 825\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 826\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfetch_result\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mDALQueryError\u001b[0m: Query Error"
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
}
],
@ -200,16 +187,539 @@
"\n",
"# Perform an ADQL query on the service, using RA and DEC from the DDRG table.\n",
"\n",
"evn_search = tap_evn.run_async(QUERY, uploads={\"ddrg\":ddrg})\n",
"evn_search = tap_evn.run_sync(QUERY, uploads={\"ddrg\":ddrg})\n",
"\n",
"# If you launch TOPCAT before running the notebook, the results will be listed there\n",
"evn_search.broadcast(\"topcat\")"
"evn_search.broadcast_samp(\"topcat\")"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "private-board",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<Table length=0>\n",
"dataproduct_type dataproduct_subtype calib_level ... Freq size FileName\n",
" ... GHz Kibyte \n",
" object object int16 ... float64 int32 object \n",
"---------------- ------------------- ----------- ... ------- ------ --------"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"evn_search"
]
},
{
"cell_type": "markdown",
"id": "residential-ticket",
"metadata": {},
"source": [
"Unfortunately, the resulting table is empty, meaning there is no EVN data within 0.1 degree of these objects. There are several options to try:\n",
"- widen the search area: trying to increase to >0.1 causes a fail\n",
"- change the order of the JOIN step: this has no effect on the results\n",
"- use the EVN FOV instead of a preset radius\n",
"\n",
"Alternatively, a query like this can be expanded to search for *any* data for these objects in all VO services. This joins the example 4 strategy of search all services with this notebook.\n",
"\n",
"Since there's no EVN data for this sample: go write a proposal!"
]
},
{
"cell_type": "markdown",
"id": "discrete-gender",
"metadata": {},
"source": [
"Try to search for othter VO data for the DDRG table"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "mysterious-census",
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:3:1106: UnknownElementWarning: Unknown element mirrorURL [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:100:4268: UnknownElementWarning: Unknown element mirrorURL [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:100:4585: UnknownElementWarning: Unknown element mirrorURL [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:100:4897: UnknownElementWarning: Unknown element mirrorURL [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:100:5197: UnknownElementWarning: Unknown element mirrorURL [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://gavo.aip.de/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:251: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:300: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: http://gavo.aip.de/tap (Query Error)\n",
"\n",
"Querying http://vao.stsci.edu/CAOMTAP/TapService.aspx\n",
" Broken: http://vao.stsci.edu/CAOMTAP/TapService.aspx ('NoneType' object has no attribute 'href')\n",
"\n",
"Querying http://astroweb.projects.phys.ucl.ac.uk:8000/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying https://vo.astron.nl/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:396: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:445: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: https://vo.astron.nl/tap (Query Error)\n",
"\n",
"Querying http://vos2.asu.cas.cz/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:364: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:413: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: http://vos2.asu.cas.cz/tap (Query Error)\n",
"\n",
"Querying https://atoavo.atnf.csiro.au/tap\n",
" Broken: https://atoavo.atnf.csiro.au/tap (Cannot parse datetime 2021-04-24T19:11:24.128+1000)\n",
"\n",
"Querying https://casda.csiro.au/casda_vo_tools/tap\n",
" Broken: https://casda.csiro.au/casda_vo_tools/tap (Cannot parse datetime 2021-04-24T17:11:29.926+0800)\n",
"\n",
"Querying http://arvo-registry.sci.am/tap\n",
" Broken: http://arvo-registry.sci.am/tap (504 Server Error: Gateway Time-out for url: http://arvo-registry.sci.am/__system__/tap/run/tap/async/XVmsB7?WAIT=-1)\n",
"\n",
"Querying https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/argus\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:25:2: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:26:4: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/argus (Query Error)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/tap/jplus-dr1?\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:28:2: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:29:2: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:31:2: UnknownElementWarning: Unknown element jobInfo [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: https://archive.cefca.es/catalogues/vo/tap/jplus-dr1? (Query Error)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/siap/jplus-dr1?\n",
" Broken: https://archive.cefca.es/catalogues/vo/siap/jplus-dr1? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr1/JPLUS.MAGABDUALOBJ?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr1/JPLUS.MAGABDUALOBJ? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr1/JPLUS.MAGABSINGLEOBJ?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr1/JPLUS.MAGABSINGLEOBJ? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr1/JPLUS.TILEIMAGE?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr1/JPLUS.TILEIMAGE? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/tap/jplus-dr2\n",
" Broken: https://archive.cefca.es/catalogues/vo/tap/jplus-dr2 (Query Error)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/siap/jplus-dr2?\n",
" Broken: https://archive.cefca.es/catalogues/vo/siap/jplus-dr2? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/siap/jplus-dr2/reduced?\n",
" Broken: https://archive.cefca.es/catalogues/vo/siap/jplus-dr2/reduced? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.MAGABDUALOBJ?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.MAGABDUALOBJ? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.TILEIMAGEWITHFLAG?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.TILEIMAGEWITHFLAG? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.XMATCH_MINIJPAS_PDR201912?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.XMATCH_MINIJPAS_PDR201912? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.XMATCH_JPLUS_DR1?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.XMATCH_JPLUS_DR1? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.MAGABSINGLEOBJ?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.MAGABSINGLEOBJ? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.REDUCEDINDIVIDUALFRAME?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.REDUCEDINDIVIDUALFRAME? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.TILEIMAGE?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/jplus-dr2/JPLUS.TILEIMAGE? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/tap/minijpas-pdr201912\n",
" Broken: https://archive.cefca.es/catalogues/vo/tap/minijpas-pdr201912 (Query Error)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/siap/minijpas-pdr201912?\n",
" Broken: https://archive.cefca.es/catalogues/vo/siap/minijpas-pdr201912? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.XMATCH_JPLUS_DR1?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.XMATCH_JPLUS_DR1? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.MAGABDUALOBJ?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.MAGABDUALOBJ? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.TILEIMAGE?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.TILEIMAGE? (123:2: mismatched tag)\n",
"\n",
"Querying https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.MAGABSINGLEOBJ?\n",
" Broken: https://archive.cefca.es/catalogues/vo/cone/minijpas-pdr201912/MINIJPAS.MAGABSINGLEOBJ? (123:2: mismatched tag)\n",
"\n",
"Querying https://vo.chivo.cl/tap\n",
" Broken: https://vo.chivo.cl/tap (index 0 is out of bounds for axis 0 with size 0)\n",
"\n",
"Querying http://archive.eso.org/tap_obs\n",
" Broken: http://archive.eso.org/tap_obs (404 Client Error: for url: http://archive.eso.org/tap_obs/async/phase)\n",
"\n",
"Querying http://dachs.planet.fu-berlin.de/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://ia2-tap.oats.inaf.it:8080/wgetap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:24:2: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:25:4: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: http://ia2-tap.oats.inaf.it:8080/wgetap (Query Error)\n",
"\n",
"Querying http://vo-node1.iaps.inaf.it/tap\n",
" Broken: http://vo-node1.iaps.inaf.it/tap (HTTPConnectionPool(host='vo-node1.iaps.inaf.it', port=80): Max retries exceeded with url: /tap/async (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb56d24b8b0>: Failed to establish a new connection: [Errno 60] Operation timed out')))\n",
"\n",
"Querying http://dorotka.ufa.cas.cz/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:278: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:327: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: http://dorotka.ufa.cas.cz/tap (Query Error)\n",
"\n",
"Querying http://jvo.nao.ac.jp/skynode/do/tap/alma\n",
" Broken: http://jvo.nao.ac.jp/skynode/do/tap/alma (404 Client Error: for url: http://jvo.nao.ac.jp/skynode/do/tap/alma/async/phase)\n",
"\n",
"Querying http://jvo.nao.ac.jp/skynode/do/tap/halca\n",
" Broken: http://jvo.nao.ac.jp/skynode/do/tap/halca (404 Client Error: for url: http://jvo.nao.ac.jp/skynode/do/tap/halca/async/phase)\n",
"\n",
"Querying http://jvo.nao.ac.jp/skynode/do/tap/hitomi\n",
" Broken: http://jvo.nao.ac.jp/skynode/do/tap/hitomi (404 Client Error: for url: http://jvo.nao.ac.jp/skynode/do/tap/hitomi/async/phase)\n",
"\n",
"Querying http://jvo.nao.ac.jp/skynode/do/tap/nobeyama/\n",
" Broken: http://jvo.nao.ac.jp/skynode/do/tap/nobeyama/ (404 Client Error: for url: http://jvo.nao.ac.jp/skynode/do/tap/nobeyama//async/phase)\n",
"\n",
"Querying http://jvo.nao.ac.jp/skynode/do/tap/spcam\n",
" Broken: http://jvo.nao.ac.jp/skynode/do/tap/spcam (404 Client Error: for url: http://jvo.nao.ac.jp/skynode/do/tap/spcam/async/phase)\n",
"\n",
"Querying http://jvo.nao.ac.jp/skynode/do/siap/spcam/image_cutout/1.0?\n",
" Broken: http://jvo.nao.ac.jp/skynode/do/siap/spcam/image_cutout/1.0? (Cannot wait for job completion. Job is not active!)\n",
"\n",
"Querying http://jvo.nao.ac.jp/skynode/do/siap/spcam/image_nocut/1.0?\n",
" Broken: http://jvo.nao.ac.jp/skynode/do/siap/spcam/image_nocut/1.0? (Cannot wait for job completion. Job is not active!)\n",
"\n",
"Querying http://machotap.asvo.nci.org.au/ncitap/tap\n",
" Broken: http://machotap.asvo.nci.org.au/ncitap/tap (Cannot parse datetime 2021-04-20T19:14:45.745+1000)\n",
"\n",
"Querying http://api.skymapper.nci.org.au/public/tap\n",
" Broken: http://api.skymapper.nci.org.au/public/tap (Cannot parse datetime 2021-04-20T19:14:49.037+1000)\n",
"\n",
"Querying http://wiggleztap.asvo.nci.org.au/ncitap/tap\n",
" Broken: http://wiggleztap.asvo.nci.org.au/ncitap/tap (Cannot parse datetime 2021-04-20T19:14:52.093+1000)\n",
"\n",
"Querying http://dc.zah.uni-heidelberg.de/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:401: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:450: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: http://dc.zah.uni-heidelberg.de/tap (Query Error)\n",
"\n",
"Querying http://osug-vo.osug.fr:8080/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://voparis-rr.obspm.fr:80/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://pvol2.ehu.eus:8083/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://spectrum.iaa.es/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying https://gaia.ari.uni-heidelberg.de/tap\n",
" Broken: https://gaia.ari.uni-heidelberg.de/tap (Cannot wait for job completion. Job is not active!)\n",
"\n",
"Querying http://epn-tap.oeaw.ac.at/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://voparis-tap-astro.obspm.fr/__system__/tap/run/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:410: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:459: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: http://voparis-tap-astro.obspm.fr/__system__/tap/run/tap (Query Error)\n",
"\n",
"Querying http://voparis-tap-maser.obspm.fr:80/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:8:269: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n",
"WARNING: UnknownElementWarning: None:8:318: UnknownElementWarning: Unknown element message [pyvo.utils.xml.elements]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" Broken: http://voparis-tap-maser.obspm.fr:80/tap (Query Error)\n",
"\n",
"Querying http://voparis-tap-voevent.obspm.fr:80/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://rcsed-vo.sai.msu.ru/tap\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: W48: ?:?:?: W48: Unknown attribute 'xtype' on FIELD [astropy.io.votable.tree]\n",
"WARNING: W28: ?:?:?: W28: 'xtype' on 'FIELD' added in VOTable 1.2 [astropy.io.votable.tree]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Querying http://xcatdb.unistra.fr/3xmmdr7/tap\n",
" Broken: http://xcatdb.unistra.fr/3xmmdr7/tap (Cannot parse datetime 2021-04-19T12:01:31.803+0200)\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: UnknownElementWarning: None:17:1: UnknownElementWarning: Unknown element errorSummary [pyvo.utils.xml.elements]\n"
]
}
],
"source": [
"# Copy and pasted from 'example 4'. This can take a while to run.\n",
"# Note that the query is inverted wrt the example, assuming all obscore tables have a FOV value\n",
"# This seems to work OK\n",
"\n",
"obsvc=pyvo.registry.search(datamodel=\"obscore\")\n",
"for row in obsvc:\n",
" \n",
" try:\n",
" print (\"Querying {url}\".format(url=row.access_url))\n",
" service=pyvo.dal.TAPService(row.access_url)\n",
" ddrg_result=service.run_async(query=QUERY, uploads={\"ddrg\":ddrg})\n",
" ddrg_result.broadcast_samp(\"topcat\")\n",
" \n",
" except Exception as msg:\n",
" # some service is broken; you *should* complain, but then let's be lazy here\n",
" print(\" Broken: {} ({})\\n\".format(row.access_url, msg))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "associate-expense",
"id": "parliamentary-marble",
"metadata": {},
"outputs": [],
"source": []

Loading…
Cancel
Save