How to make OpenIndiana work on RTX 30 series nVidia GPUs.

Post your tutorials for using applications or performing related tasks here.
Note: Not for "how do I...?" Questions!
Forum rules
Tutorials and Howtos should only relate to developed software, and not to third party applications. e.g.: Don't post a generic Howto for configuring a firewall.
If you have a question how to do something, you should use one of the support boards, not this board. It is meant for people to document and post instructions.
User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1537
Joined: 2018-10-28, 19:56
Location: Georgia

How to make OpenIndiana work on RTX 30 series nVidia GPUs.

Unread post by athenian200 » 2023-11-11, 01:14

I just tried some of the latest images, and apparently the OI team broke the GUI installer again so that it installs old drivers that don't work on newer GPUs. I thought this issue was fixed the last time I checked, but apparently they somehow broke it again, probably because their fix for the previous issue caused upgrade installs to break on existing systems, and now they've recaused the original issue fixing that one. This tutorial was tested and found to work on an RTX 3060, your milage may vary with an RTX 20 series, but I imagine it would work for those cards too. I have no idea about RTX 40 series, though.

As a result of this, a lot of people otherwise willing to help me out with SunOS builds of Pale Moon are unable to figure out how to get it working. So it's only tangentially related to the project, but hopefully you don't mind me posting this here, since OI doesn't have its own forum and getting info like this out into a place where it will be seen is otherwise difficult.

1. Download the text mode installer of OpenIndiana, not the GUI/Live CD version. That one will load in the wrong drivers and you'll have a bad time.

2. Follow the prompts and reboot as needed.

3. Login from the console.

4.

Code: Select all

sudo pkg image-update -v
is needed before you start, otherwise you may not be able to pull packages from the OI repo due to weird version conflicts, as they pull older versions of packages all the time.

You will likely be asked to reboot, make sure you do so.

6.

Code: Select all

sudo pkg install mate_install
7.

Code: Select all

sudo pkg uninstall mate_install
Explanation: mate_install is a meta package, and installing it installs a whole bunch of stuff, and locks you into specific versions of specific packages. Uninstalling it leaves most of those packages in place, but unlocks everything so you can remove packages as you please. We need to do this to be able to uninstall the video drivers that were just installed, because they are the wrong version.

8.

Code: Select all

sudo pkg list | grep nvidia
Look at what driver is installed. More than likely it's nvidia-340 or nvidia-390, neither of which work with the RTX 30 series cards. nvidia-470 used to work for me, but didn't work as I was writing this tutorial. The driver you want is "nvidia" and should have a number starting with 500 or so. I use nvidia-340 as the example, but you may have to use nvidia-390 or nvidia-470 because it seems to be random which driver version gets picked.

9.

Code: Select all

sudo pkg uninstall xorg-video nvidia-340
10.

Code: Select all

sudo pkg install nvidia
After rebooting (again), try doing step 8 again. You should see something like:

Code: Select all

driver/graphics/nvidia									535.129.3-2023.0.0.0	i--
Seeing text that mentions the nVidia modesetting driver is a very good sign you're on the right path.

11.

Code: Select all

sudo svcadm enable lightdm
This command should work if you did everything else correctly, and bring you to the desktop. I have no idea why they broke this again, but it was working last year I think. I think too many people on OI are using older nVidia cards and unfortunately they haven't found a good way to make it work out of the box on both older and newer GPUs by detecting the GPU and then applying the correct driver. Whatever detection they are trying to use seems to work wrong, so the only real solution is to manually install the desktop package from a text install, and then pick the nVidia driver you need rather than letting them pick one for you.

Anyway, after that, to build Pale Moon, you would (at least try to) follow the directions here (a bit out of date, I need to fix them):

https://developer.palemoon.org/build/sunos/

And since Python 2.x is no longer supported, we may also need to adapt Step 2 from this guide:

viewtopic.php?f=19&t=30532

Which relates to building Python 2 from source rather than relying on having it packaged. EDIT: Nevermind, Python 2 appears to be back in the repos?

NOTE: This is really more something I want to have here so I can reference it wherever I am easily.
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind