Upgrade MB, CPU & Memory

First off unless it's new I'd suggest also replacing the PSU, it's presumably closing in on 10 years old, came with a 5 year warranty, it's mean time between failure is about 11.5 years 24/7 and it doesn't seem to have an 80+ certification. Obviously you don't have to but I'd say it's time.

Second I think I'd personally go with an Intel cpu for quicksync and also AMD have just recently put the (Vega) graphics in their AM4 apus (e.g. 5600G) onto life support in terms of driver development so it feels really ugly to buy one of them now to use those graphics. I'm pretty sure at this point that quicksync would be better than your 1050ti. The 1050ti would be a better graphics card in general though but not sure you would care with the use case. You could use the 1050ti but still use quicksync for the video encoding.

You don't say anything about price etc so shooting in the dark maybe something like:

CPU: Intel Core i3-12100 3.3 GHz Quad-Core Processor (€113.88 @ Amazon Deutschland)

Motherboard: Gigabyte B760M DS3H DDR4 Micro ATX LGA1700 Motherboard (€111.90 @ Amazon Deutschland)

Memory: Corsair Vengeance LPX 16 GB (2 x 8 GB) DDR4-3600 CL18 Memory (€38.90 @ Amazon Deutschland)

Power Supply: EVGA 650 BQ 650 W 80+ Bronze Certified Semi-modular ATX Power Supply (€63.02 @ Amazon Deutschland)

Total: €327.70 (plus VAT change and delivery)

There's certainly cheaper boards, I picked that one as it has a 2.5Gb nic (if your NAS has it or can be upgraded that might be an easy worthwhile quality of life improvement) and it also has sane video outputs as opposed to some of the cheapest which are have 1 vga and 1 hdmi. If you don't care about the 2.5GB nic or pushing the ram up to 3600 instead of stock 3200, or 2 dimm slots or pretty much anything except budget you could get a ASRock H610M-HDV for about €75.

Board also has 4 Dimm slots so easy to pop in more ram, given you ran the old system for nearly 10 years I'd go with 32GB if you were getting a 2 Dimm board as you could get 2x16GB of 3200CL16 for €65 or 3600CL18 for €72, so cheaper per GB than 2x8. If the budget isn't really bothering you just get 2x16GB or 3600CL18 anyway now.

I got bored hunting for a sane manufacturers PSU in stock from Amazon Germany themselves so it's not the cheapest but is semi-modular for less cable mess. Depending how much you run the PC and your tolerance for noise you could spend more on an even more efficient PSU and/or one with fan-stop which probably wouldn't ever spin it's fan here.

Again depending on your tolerance to noise you could add a something like a €20 Thermalright cpu cooler.

Clearly you could spend more on a beefier cpu and slightly faster graphics but it doesn't sound like you need to. A 12600K or 13500 are roughly €250 and the cheapest with the slightly better graphics, a 12400 is about €170 for a hex-core with the same graphics.

You could also go AM5 at that point, a Ryzen 7600 should be just over €200, the board maybe €130 (again for 2.5GBit and not really minimal video outputs etc) and ram ballpark €125 (I'd go 6000CL30 for €135 as it's the performance sweet spot anyway and when you are using the onboard graphics I'd spend the €20 extra over CL36). That's about an extra €200 and massive overkill for future proofing I doubt you'll use and quicksync is more likely to be properly tuned to reliably work with anything whereas ymmv with the Ryzen's video encoding.

This is from July last year. Note some things like the "improved h264 encoding" for AMD is Navi+ which means NOT for the AM4 apus like the 5600g. The 7600 should have the improved performance but again note the video starts with explaining how it look a long time to test the improvements as it took quite a while for the software improvements from AMD to make it into any actual software people use.

After the intro you could skip to 8:43 to where the comparisons really start, but remember he's comparing the modern stuff so not your 1050ti or the AM4 chips, there was a chart in the intro early on to give you some kind of idea about those. Or 10:19 to get straight to the real punchlines.

Of course depending what you really care about maybe a modern graphics card would make you happiest to encode to AV1, sadly it's not as easy as it should be to get one of the cheap Intel cards (A310, A380) over here at the price they should be (maybe €120 though they can be $100 in the USA) and the cheapest nvidia (4060) and AMD cards (7600) which will do it are around €300!