Poor frame pacing means you shouldn't play Forza with an uncapped framerate. 

One week after its debut on PC and Xbox One, Forza Motorsport 7 continues to leave me baffled when it comes to performance. If this is the promised DX12 land of milk and honey, I'm not buying it. Microsoft and its partner studios have a lot of work to do, with performance that is consistently inconsistent. I had initially hoped to have a full performance analysis up by today, but I ran into anomalies that delayed things as I tested and retested on a variety of cards and systems. And then AMD and Nvidia just released new drivers yesterday, and at least the Nvidia drivers have invalidated all my existing results.

But I digress. Here's what I've found so far.

Performance in Forza Motorsport 7 fluctuates wildly when you're racing, dropping from framerates of nearly 200 fps into the sub-60 fps range. These drops appear to be somewhat random, though they occur repeatedly. If you run a short 30 second benchmark, like the built-in test, sometimes you'll get lucky and a drop won't occur. More often, however, performance will swan dive for a few seconds before going back high… and then swan diving again. Let me illustrate the problem with two cards, the GTX 1080 Ti and the RX Vega 64.


The charts above show performance across a number of tests on the same in-game benchmark. You can see that the same card, tested multiple times at each setting, delivers dramatically different results. The 1080 Ti has some massive swings in minimum fps, particularly with the new 387.92 drivers at 1080p medium, but even at 1440p and 4k ultra there are still fluctuations. AMD's not immune to the problems, and at least on the i7-8700K tests I ran, minimum fps was consistently low at 1080p medium, with higher minimum performance at 1080p max. Interestingly, RX Vega 64 did much better on a Core i3-7350K or a Ryzen 3 1300X in terms of minimum framerates, but that's mostly due to the other inconsistencies.

Here's another view of what's happening on three individual benchmark runs for the Vega 64 and 1080 Ti. This time, all of the results are at 1080p ultra settings, with the Vega 64 running on Ryzen 7 1800X and the 1080 Ti running on Core i7-8700K. The point isn't to compare AMD vs. Nvidia, but rather look at each individual run, and notice how inconsistent the frametimes can be. (You'll need to click for the fullsize images, as these are very wide in order to preserve detail, showing 27 seconds of framerate data.)


There are a lot of frame pacing issues, with a constant zigzag pattern emerging in all the frametime captures I've done. Ideally, we should be seeing more of a smooth line that may curve up or down over time, but without any massive spikes. I'm not clear on whether the problem is in the game or in the AMD and Nvidia drivers, but since it's a DX12 engine the responsibility for frame pacing should fall more on the shoulders of the game developers.

I haven't had a chance to (re)test all the other GPUs, but suffice it to say performance is at best wonky. From what I can tell, hitting higher framerates causes the larger drops in minimum fps. The closer the game gets to a steady 60 fps, the more consistent the minimum framerates become. If you're playing or hoping to play Forza Motorsport 7 on Windows 10 anytime soon, my advice is to use the game's 60 fps framerate cap—which sucks if you're on a 144Hz display, but it sucks less than the variations in performance I'm currently seeing.


As far as what hardware you'll need to run the game, there are some pleasant surprises. GTX 1080 and 1080 Ti are able to hit 60 fps at 4k with maxed out settings and 4xMSAA, along with the RX Vega 56 and 64. GTX 1070 is also right on the 60 fps threshold, and with the latest drivers it should exceed that mark. Even lesser cards like RX 580/570 and GTX 1060 6GB get pretty close to 60 fps, and a few tweaks to the settings should push them over the hump, but be wary of cards with 4GB or less VRAM, as performance can take a nosedive at higher quality settings compared to cards with 6GB or more VRAM. For budget cards at the level of the GTX 1050 or RX 560, 1080p medium to high quality should get you to 60 fps as well, and older generation hardware doesn't seem to have any major performance penalties.

I reached out to Turn 10 Studios regarding the inconsistencies I encountered, and received this official statement:

"Since the release of 'Forza Motorsport 7,' stability and performance continue to be top priorities for the Windows 10 PC version. We’ve made two updates to the game so far, addressing multiple stability issues, and future updates will include improvements to unlocked modes. We’re continuing to investigate and solve issues as they arise, with the goal of giving all 'Forza Motorsport 7' players a smooth experience on Windows 10 PCs." – Turn 10 Studios

Let's hope things improve sooner than later, as there are plenty of bugs and other issues that need fixing. You can read the official notes on previous updates at ForzaMotorsport.net.


There's a separate issue that's impeded my testing as well, because Forza Motorsport 7 is the worst offender I've encountered so far. More than two years after the release release of Windows 10, the Windows Store is still an awful way to distribute games. Sure, it will download a game just fine. But let's say you have two PCs, like a laptop and a desktop, or an HTPC and an office PC. The way Windows Store works is that you download the application twice. Period. DRM isn't going to have it any other way.

That's not too bad when you're looking at the Windows Store version of Candy Crush or some other application that's a few hundred MB at most. Unfortunately, Forza Motorsport 7 tips that scales at 102.8GB. Even on a relatively fast 250Mbps connection that requires about an hour to download, and on a 50Mbps connection you're looking at five hours. I had to do that on the three PCs I used in my initial testing—which adds up quickly when I'm looking at the 1TB Comcast data cap.

Of course, not many people are downloading a game multiple times to test hardware like I am. But if something bad were to happen and you need to reinstall your OS, there's no way to point the Windows Store at an existing folder and say, "Please verify all the files I've already downloaded." In fact, you can't even view the files in Explorer by default as they're owned by the TrustedInstaller service. You can take ownership and grant yourself access rights, but that can have unintended consequences (as in, things break if the permissions for the WindowsApps folder change).


The point is, Steam has supported checking downloaded files practically since its inception, and Origin, GoG Galaxy, and UPlay also allow importing/verifying an existing folder so that you don't have to download all the files again. The Windows Store has no equivalent functionality, which makes it the worst gaming distribution service, which becomes more frustrating as game download sizes continue to balloon. With Forza around 100GB and Middle-earth: Shadow of War right there as well, this is something Microsoft should look to address post-haste. An option to have the Windows Store copy files over the network from another PC would be a cool bonus, but just the ability to backup/restore Windows Store apps is sorely needed.

I could say we shouldn't be greedy—getting Microsoft's first-party games for both Windows and Xbox One as a package is pretty cool. Or at least it is if you own an Xbox One. But while the Xbox Play Anywhere service sounds great, saddling it with the Windows Store in its current implementation is hurting the experience of gaming on Windows 10.
Hardware

Unlocked framerates on Forza Motorsport 7 are wildly inconsistent

Poor frame pacing means you shouldn't play Forza with an uncapped framerate. 

One week after its debut on PC and Xbox One, Forza Motorsport 7 continues to leave me baffled when it comes to performance. If this is the promised DX12 land of milk and honey, I'm not buying it. Microsoft and its partner studios have a lot of work to do, with performance that is consistently inconsistent. I had initially hoped to have a full performance analysis up by today, but I ran into anomalies that delayed things as I tested and retested on a variety of cards and systems. And then AMD and Nvidia just released new drivers yesterday, and at least the Nvidia drivers have invalidated all my existing results.

But I digress. Here's what I've found so far.

Performance in Forza Motorsport 7 fluctuates wildly when you're racing, dropping from framerates of nearly 200 fps into the sub-60 fps range. These drops appear to be somewhat random, though they occur repeatedly. If you run a short 30 second benchmark, like the built-in test, sometimes you'll get lucky and a drop won't occur. More often, however, performance will swan dive for a few seconds before going back high… and then swan diving again. Let me illustrate the problem with two cards, the GTX 1080 Ti and the RX Vega 64.


The charts above show performance across a number of tests on the same in-game benchmark. You can see that the same card, tested multiple times at each setting, delivers dramatically different results. The 1080 Ti has some massive swings in minimum fps, particularly with the new 387.92 drivers at 1080p medium, but even at 1440p and 4k ultra there are still fluctuations. AMD's not immune to the problems, and at least on the i7-8700K tests I ran, minimum fps was consistently low at 1080p medium, with higher minimum performance at 1080p max. Interestingly, RX Vega 64 did much better on a Core i3-7350K or a Ryzen 3 1300X in terms of minimum framerates, but that's mostly due to the other inconsistencies.

Here's another view of what's happening on three individual benchmark runs for the Vega 64 and 1080 Ti. This time, all of the results are at 1080p ultra settings, with the Vega 64 running on Ryzen 7 1800X and the 1080 Ti running on Core i7-8700K. The point isn't to compare AMD vs. Nvidia, but rather look at each individual run, and notice how inconsistent the frametimes can be. (You'll need to click for the fullsize images, as these are very wide in order to preserve detail, showing 27 seconds of framerate data.)


There are a lot of frame pacing issues, with a constant zigzag pattern emerging in all the frametime captures I've done. Ideally, we should be seeing more of a smooth line that may curve up or down over time, but without any massive spikes. I'm not clear on whether the problem is in the game or in the AMD and Nvidia drivers, but since it's a DX12 engine the responsibility for frame pacing should fall more on the shoulders of the game developers.

I haven't had a chance to (re)test all the other GPUs, but suffice it to say performance is at best wonky. From what I can tell, hitting higher framerates causes the larger drops in minimum fps. The closer the game gets to a steady 60 fps, the more consistent the minimum framerates become. If you're playing or hoping to play Forza Motorsport 7 on Windows 10 anytime soon, my advice is to use the game's 60 fps framerate cap—which sucks if you're on a 144Hz display, but it sucks less than the variations in performance I'm currently seeing.


As far as what hardware you'll need to run the game, there are some pleasant surprises. GTX 1080 and 1080 Ti are able to hit 60 fps at 4k with maxed out settings and 4xMSAA, along with the RX Vega 56 and 64. GTX 1070 is also right on the 60 fps threshold, and with the latest drivers it should exceed that mark. Even lesser cards like RX 580/570 and GTX 1060 6GB get pretty close to 60 fps, and a few tweaks to the settings should push them over the hump, but be wary of cards with 4GB or less VRAM, as performance can take a nosedive at higher quality settings compared to cards with 6GB or more VRAM. For budget cards at the level of the GTX 1050 or RX 560, 1080p medium to high quality should get you to 60 fps as well, and older generation hardware doesn't seem to have any major performance penalties.

I reached out to Turn 10 Studios regarding the inconsistencies I encountered, and received this official statement:

"Since the release of 'Forza Motorsport 7,' stability and performance continue to be top priorities for the Windows 10 PC version. We’ve made two updates to the game so far, addressing multiple stability issues, and future updates will include improvements to unlocked modes. We’re continuing to investigate and solve issues as they arise, with the goal of giving all 'Forza Motorsport 7' players a smooth experience on Windows 10 PCs." – Turn 10 Studios

Let's hope things improve sooner than later, as there are plenty of bugs and other issues that need fixing. You can read the official notes on previous updates at ForzaMotorsport.net.


There's a separate issue that's impeded my testing as well, because Forza Motorsport 7 is the worst offender I've encountered so far. More than two years after the release release of Windows 10, the Windows Store is still an awful way to distribute games. Sure, it will download a game just fine. But let's say you have two PCs, like a laptop and a desktop, or an HTPC and an office PC. The way Windows Store works is that you download the application twice. Period. DRM isn't going to have it any other way.

That's not too bad when you're looking at the Windows Store version of Candy Crush or some other application that's a few hundred MB at most. Unfortunately, Forza Motorsport 7 tips that scales at 102.8GB. Even on a relatively fast 250Mbps connection that requires about an hour to download, and on a 50Mbps connection you're looking at five hours. I had to do that on the three PCs I used in my initial testing—which adds up quickly when I'm looking at the 1TB Comcast data cap.

Of course, not many people are downloading a game multiple times to test hardware like I am. But if something bad were to happen and you need to reinstall your OS, there's no way to point the Windows Store at an existing folder and say, "Please verify all the files I've already downloaded." In fact, you can't even view the files in Explorer by default as they're owned by the TrustedInstaller service. You can take ownership and grant yourself access rights, but that can have unintended consequences (as in, things break if the permissions for the WindowsApps folder change).


The point is, Steam has supported checking downloaded files practically since its inception, and Origin, GoG Galaxy, and UPlay also allow importing/verifying an existing folder so that you don't have to download all the files again. The Windows Store has no equivalent functionality, which makes it the worst gaming distribution service, which becomes more frustrating as game download sizes continue to balloon. With Forza around 100GB and Middle-earth: Shadow of War right there as well, this is something Microsoft should look to address post-haste. An option to have the Windows Store copy files over the network from another PC would be a cool bonus, but just the ability to backup/restore Windows Store apps is sorely needed.

I could say we shouldn't be greedy—getting Microsoft's first-party games for both Windows and Xbox One as a package is pretty cool. Or at least it is if you own an Xbox One. But while the Xbox Play Anywhere service sounds great, saddling it with the Windows Store in its current implementation is hurting the experience of gaming on Windows 10.

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)