Dublin Bus RTPI & Home Assistant

dregin · 25-10-2022 3:05pm #1

Hey folks,

I've taken a look around and can't find much discussion on the use of api.nationaltransport.ie - I'm new to Home Assistant and am trying to configure https://github.com/zacs/ha-gtfs-rt to use the NTA's API for bus information.

My HA sensor is configured as follows

- platform: gtfs_rt
  trip_update_url: "https://api.nationaltransport.ie/gtfsr/v1"
  x_api_key: "REDACTED"
  departures:
    - name: "38 from Parnell Square"
      route: "60-38-b12-1"
      stopid: "8220DB000002"

I took the route and stop ID's from https://transitfeeds.com/p/transport-for-ireland/782/latest/stop/8220DB000002

All I get on the dashbaord card in HA is - min for each sensor I've configured.

Logs have nothing. I tried to enable debug logs for the integration as described in the README, to no avail.

Thanks!

niallb · 25-10-2022 5:39pm

Nice find.

Where did you get the API key you're using?

From what I'm seeing there in a post from June 10th 2022 , the v1 api is being retired "in the coming months".

If you think you have a valid API key, could you try with this trip_update_url instead ?

https://gtfsr.transportforireland.ie/v1

niallb · 25-10-2022 6:03pm

Have you tried it with stopid '1' ? The stop number on the route rather than the stop ID number?

Nope. You were right

"stop_id": "Must be the same as in stops.txt in the corresponding GTFS feed e.g. 8430B2552301",

Worth trying stop_id rather than stopid though.

I'm getting Auth errors logged for the call so haven't yet spotted the full string.

dregin · 25-10-2022 8:58pm

https://www.boards.ie/discussion/comment/119792450#Comment_119792450

Thanks, I'd already tried with stop_id , but the integration rejects it:

dregin · 25-10-2022 8:59pm

https://www.boards.ie/discussion/comment/119792340#Comment_119792340

Aye, I tried the new URL, but it returns a 404.

I got the API by signing up here: https://developer.nationaltransport.ie/apis

dregin · 25-10-2022 9:07pm

https://www.boards.ie/discussion/comment/119792340#Comment_119792340

I've tried that URL. Seeing no complaints in the logs, but also no updates on the dashboard card.

Thanks a mill for all the info btw.

dregin · 25-10-2022 9:11pm

Oddly enough, I do see this for one of the routes, so it looks like it got at least some data.

niallb · 26-10-2022 9:26am

I'd love to get this working, so happy to stick with it until we do!

I got my API key in the same way, but I was getting this returned in the logs every time.

updating route status got 403:b'{"message":"Invalid API Key"}'

I just changed the URL I'm using to trip_update_url: "https://api.nationaltransport.ie/gtfsr/v1[?format]"

which is written on the API description in the documentation and now I'm seeing this instead:

updating route status got 404:b'{ "statusCode": 404, "message": "Resource not found" }'

I suspect we have to add the contents of the downloadable zip file with stops.txt, routes.txt and the rest of it somewhere in the filesystem on home assistant but I haven't worked out where to put it if that's needed.

It's the one at https://www.transportforireland.ie/transitData/google_transit_combined.zip

dregin · 26-10-2022 11:06am

Are you using X_API_KEY in your config?

emaherx · 26-10-2022 3:45pm

Have you guys tried exploring the API with Node-Red?

Just import the following and add your API-Key to the header section of the http request node.

[
    {
        "id": "b9eb633e51f8ccf4",
        "type": "http request",
        "z": "a433ad12c9378bd7",
        "name": "",
        "method": "GET",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "https://api.nationaltransport.ie/gtfsr/v1?format=json",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [
            {
                "keyType": "Cache-Control",
                "keyValue": "",
                "valueType": "no-cache",
                "valueValue": ""
            },
            {
                "keyType": "other",
                "keyValue": "x-api-key",
                "valueType": "other",
                "valueValue": "yourAPIkeyHERE"
            }
        ],
        "credentials": {},
        "x": 370,
        "y": 280,
        "wires": [
            [
                "96a69b654e0a7c21"
            ]
        ]
    },
    {
        "id": "dad9265666b90db2",
        "type": "inject",
        "z": "a433ad12c9378bd7",
        "name": "",
        "props": [],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 210,
        "y": 280,
        "wires": [
            [
                "b9eb633e51f8ccf4"
            ]
        ]
    },
    {
        "id": "96a69b654e0a7c21",
        "type": "debug",
        "z": "a433ad12c9378bd7",
        "name": "debug 3",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 540,
        "y": 280,
        "wires": []
    }
]

Jonathan · 26-10-2022 5:11pm

This thread is likely of interest.

dregin · 26-10-2022 7:28pm

https://www.boards.ie/discussion/comment/119796360#Comment_119796360

Cheers, I'd seen that thread already and pretty much discounted it because it was old. The user hacking the code got me thinking. I took a look there and realized that although they tell you how to enable debug logs, the code doesn't actually emit any debug log events :facepalm:

Have shoved a load of em in now to get a better idea of what's going on. I'm definitely getting valid data from the API, by the look of things.

dregin · 26-10-2022 7:56pm

https://www.boards.ie/discussion/comment/119796360#Comment_119796360

ahhh, I see the other guy who rewrote the integration. Any idea where the index can be created? Assumed I'd find an sqlite file in my config directory, but no sign. Thanks!

emaherx · 26-10-2022 9:11pm

Route Filtered to 60-38-b12-1

emaherx · 26-10-2022 9:43pm

Route and Stop filtered, needs a little more tidying up to make a sensor, but can work if you use the Node-Red addon.

[
    {
        "id": "b9eb633e51f8ccf4",
        "type": "http request",
        "z": "a433ad12c9378bd7",
        "name": "",
        "method": "GET",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "https://api.nationaltransport.ie/gtfsr/v1?format=json",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [
            {
                "keyType": "Cache-Control",
                "keyValue": "",
                "valueType": "no-cache",
                "valueValue": ""
            },
            {
                "keyType": "other",
                "keyValue": "x-api-key",
                "valueType": "other",
                "valueValue": "YourAPIkeyHere"
            }
        ],
        "x": 590,
        "y": 120,
        "wires": [
            [
                "4a9ef200d75e9b1b"
            ]
        ]
    },
    {
        "id": "dad9265666b90db2",
        "type": "inject",
        "z": "a433ad12c9378bd7",
        "name": "",
        "props": [],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 430,
        "y": 120,
        "wires": [
            [
                "b9eb633e51f8ccf4"
            ]
        ]
    },
    {
        "id": "96a69b654e0a7c21",
        "type": "debug",
        "z": "a433ad12c9378bd7",
        "name": "debug 3",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 940,
        "y": 160,
        "wires": []
    },
    {
        "id": "4a9ef200d75e9b1b",
        "type": "function",
        "z": "a433ad12c9378bd7",
        "name": "Filter Route",
        "func": "let result = [];\nmsg.payload.Entity.forEach(dat => {\n    if (dat.Id.includes(\"60-38-b12-1\")) {\n        result.push(dat)\n    }\n})\nmsg.payload = result;\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 770,
        "y": 120,
        "wires": [
            [
                "e933666e671ec46a"
            ]
        ]
    },
    {
        "id": "d05ba899b7280ad9",
        "type": "function",
        "z": "a433ad12c9378bd7",
        "name": "Filter Stop ",
        "func": "let result = [];\nmsg.payload.TripUpdate.StopTimeUpdate.forEach(dat => {\n    if (dat.StopId.includes(\"8220DB000002\")) {\n        result.push(dat)\n    }\n})\nmsg.payload = result;\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 770,
        "y": 160,
        "wires": [
            [
                "96a69b654e0a7c21"
            ]
        ]
    },
    {
        "id": "e933666e671ec46a",
        "type": "split",
        "z": "a433ad12c9378bd7",
        "name": "",
        "splt": "\\n",
        "spltType": "str",
        "arraySplt": 1,
        "arraySpltType": "len",
        "stream": false,
        "addname": "",
        "x": 930,
        "y": 120,
        "wires": [
            [
                "d05ba899b7280ad9"
            ]
        ]
    }
]

emaherx · 27-10-2022 11:51am

@dregin does that route by any chance change throughout the day? It seems that stop is not always included in the data for that route.

dregin · 27-10-2022 8:27pm

No, it doesn't change 😔

emaherx · 27-10-2022 8:32pm

Maybe try a bus stop before or after on the same route?

dregin · 28-10-2022 6:45pm

https://www.boards.ie/discussion/comment/119801721#Comment_119801721

Pretty sure it's not that. Gonna take a stab at a python script to print out the times locally and use that as a starting point for a fork of the integration I'm trying to use.

Step 1, check their docs... here's the sample python code - imports json and proceeds to not use it at all. Great.

dregin · 28-10-2022 7:19pm

Check out the lad slagging someone else's docs while posting his API key publicly :D

Regenerated...

emaherx · 28-10-2022 8:47pm

https://www.boards.ie/discussion/comment/119805771#Comment_119805771

Easily done.

dregin · 28-10-2022 9:23pm

Right, decided to take a completely different route at this. Found the request payload used by this page and hacked it into this:

python3 get_buses.py 8220DB000002
2022-10-28T21:16:00Z: 38A
2022-10-28T21:13:00Z: 46A
2022-10-28T21:01:00Z: 38
2022-10-28T20:58:00Z: 46A
2022-10-28T20:46:00Z: 38A
2022-10-28T20:43:00Z: 46A
2022-10-28T20:33:00Z: 46A
2022-10-28T20:31:00Z: 38
2022-10-28T20:24:00Z: 46A
2022-10-28T20:23:00Z: 46A

source is:

import json, os, rapidjson, requests, sys

stop_id = str(sys.argv[1])
try:
  s = requests.session()
  response = s.get("https://journeyplanner.transportforireland.ie/nta/XML_DM_REQUEST",
    params={
      "language": "en",
      "std3_suggestMacro": "std3_suggest",
      "std3_commonMacro": "dm",
      "includeCompleteStopSeq": "1",
      "mode": "direct",
      "useAllStops": "1",
      "type_dm": "any",
      "nameInfo_dm": stop_id,
      "outputFormat": "rapidJSON"
    },
    headers={
      "DNT": "1",
      "Sec-GPC": "1"
    }
  )
  data = json.loads(response.content)

  updates = []
  for item in data['stopEvents']:
    updates.append(f"{item['departureTimePlanned']}: {item['transportation']['number']}")

  for update in sorted(updates, reverse=True):
    print(update)
except Exception as e:
  exc_type, exc_obj, exc_tb = sys.exc_info()
  fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
  print(exc_type, fname, exc_tb.tb_lineno)

dregin · 28-10-2022 9:40pm

Times are in utc and don't quite match up with next Dublin bus app, so needs tweaking.

emaherx · 29-10-2022 6:54am

https://www.boards.ie/discussion/comment/119806334#Comment_119806334

That's a fairly minor issue but well done.

It's good progress. Probably a better route too you are looking at all of the busses for that stop now rather than just the one route/stop which will obviously generate more hits, yet you can still distinguish the bus.

emaherx · 29-10-2022 8:03am

Just going to share the same method but using Node-Red which may be useful for others also, the returned json is much easier to work with than the official API, so it's a great find, thanks again @dregin

Just change stop variable in the function node for different stops (any TFI stop in the country not just Dublin bus)

[
    {
        "id": "08372f518b3d1694",
        "type": "http request",
        "z": "a433ad12c9378bd7",
        "name": "",
        "method": "GET",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [],
        "x": 470,
        "y": 300,
        "wires": [
            [
                "dab49128d865ddae"
            ]
        ]
    },
    {
        "id": "62e112536e6dfa51",
        "type": "function",
        "z": "a433ad12c9378bd7",
        "name": "function 2",
        "func": "var stop_id = 189001;\nmsg.url = \"https://journeyplanner.transportforireland.ie/nta/XML_DM_REQUEST?language=en&std3_suggestMacro=std3_suggest&std3_commonMacro=dm&includeCompleteStopSeq=1&mode=direct&useAllStops=1&type_dm=any&nameInfo_dm=\" + stop_id + \"&outputFormat=rapidJSON\";\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 300,
        "y": 300,
        "wires": [
            [
                "08372f518b3d1694"
            ]
        ]
    },
    {
        "id": "a740424328f70a4d",
        "type": "inject",
        "z": "a433ad12c9378bd7",
        "name": "",
        "props": [],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 130,
        "y": 300,
        "wires": [
            [
                "62e112536e6dfa51"
            ]
        ]
    },
    {
        "id": "dab49128d865ddae",
        "type": "debug",
        "z": "a433ad12c9378bd7",
        "name": "debug 6",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 640,
        "y": 300,
        "wires": []
    }
]

emaherx · 29-10-2022 8:55am

Sensors in Home Assistant, needs some time formatting to make a more human readable and correct time zone but it works.

dregin · 29-10-2022 3:08pm

Nice!

I'm going to dive into the deep end and try fixing https://github.com/home-assistant/core/tree/dev/homeassistant/components/dublin_bus_transport

Hold my beer...

emaherx · 30-10-2022 8:16am

Good luck, should be straight forward enough with what you are armed with now.

One suggestion would be to change the name to include TFI rather than Dublin Bus as it should work for all TFI routes if you are using the TFI site as data source. Perhaps that would involve submitting it as a new integration.

I see its actually had updates in the last month are you sure it's not working now (or perhaps the next release)?

dregin · 30-10-2022 2:06pm

I think those were find/replace jobs to address core changes. Certainly still using a URL that doesn't exist for the API.

dregin · 02-11-2022 9:14pm

Lost patience trying to fix the sensor and ended up with this:

- platform: rest
  name: "Dublin Bus"
  resource: "https://journeyplanner.transportforireland.ie/nta/XML_DM_REQUEST"
  method: GET
  headers:
    DNT: "1"
    Sec-GPC: "1"
  params:
    language: "en"
    std3_suggestMacro: "std3_suggest"
    std3_commonMacro: "dm"
    includeCompleteStopSeq: "1"
    mode: "direct"
    useAllStops: "1"
    type_dm: "any"
    nameInfo_dm: "ROUTE ID CHANGE THIS"
    outputFormat: "rapidJSON"

  value_template: "
    {%- for entry in value_json['stopEvents'] | sort(attribute='departureTimePlanned') %}
      {%- if loop.index <= 5 %}
        {%- set dateTimeExpected = strptime(entry.departureTimePlanned, '%Y-%m-%dT%H:%M:%SZ') %}
        {%- set minutes = (dateTimeExpected.replace(tzinfo=None) - utcnow().replace(tzinfo=None)).total_seconds() / 60  %}
        {{ '{:02} mins: {} <br>'.format(int(minutes),  entry.transportation.number) }}
      {%- endif %}
    {%- endfor %}
  "

evnvc · 10-02-2023 4:27pm

Hi @dregin, thank you for your rest sensor code, works as intended. Noticing a huge discrepancy with Google Maps data and the planned timetables on journey planner, do you get the same?

Another q, how do you format the data from the sensor? I just managed to include it in a html-card, but not really satisfied with the visual aspect. Did you manage to break it somehow, to include customization?

Also, final q, how should I interpret the data when the time starts with a minus?

TIA!

Edit: the sensor provides the scheduled times, doesn't update with the real time data in the Journey Planner app :(

Dublin Bus RTPI & Home Assistant

Comments