I have encountered this as well after upgrading from Spring Boot 2 to Spring Boot 3 (which upgraded logback from version 1.2.11 to 1.4.5) in one of my projects and done some analysis.  I'll share it here.

I was using ServerSocketAppender in a configuration like this on the server:

...
<appender name="LOGNET" class="ch.qos.logback.classic.net.server.ServerSocketAppender">
    <port>4560</port>
</appender>
...

With Logback 1.2.11 everything worked fine - the server started up and our client could connect and receive log messages without problems.  With Logback 1.4.5, the client would never receive any log messages, and would block when initializing the ObjectInputStream - so no bytes were being sent across the socket, not even the serialization stream headers which would have been sent under the old version when the ObjectOutputStream would have been constructed on the server. 

 Debugging the server and appender showed that under 1.4.5, connections were being accepted (in ServerSocketListener.acceptClient() from ConcurrentServerRunner.run()) but the executor that was supposed to run the RemoteReceiverStreamClient was never running it despite being asked to.  Looks like in 1.4.5, the executor (a ScheduledThreadPoolExecutor) has a pool size of 1 while in 1.2.11 it had a pool size of 10.  This executor is also being used for the thing that accepts connections from the ServerSocket, so there's no more room to actually do anything with the client connections themselves.

The executor is being created/configured from Context.getScheduledExecutorService() which, in the default implementation of ContextBase goes to ExecutorServiceUtil.newScheduledExecutorService() which gets the pool size from CoreConstants.SCHEDULED_EXECUTOR_POOL_SIZE which is currently set to 1, but used to be set to 8. I believe this is the root cause of ServerSocketAppender no longer working. The commit reduce core poll size to 1, disable cetrtain ServerSocket tests was the change.

It also looks like ServerSocketAppenderBaseFunctionalTest was disabled/ignored which now blocks indefinitely because of this bug and runs again if the pool size is changed back.

To work around the issue in my own project, I subclassed ServerSocketAppender and overrode setContext(), substituting my own wrapper implementation of Context that delegated to the original with the exception of getScheduledExecutorService() where I created my own executor service with a larger pool size. It was an ugly hack but did the job.

I'm not sure the reason of reducing the pool size and as to whether reverting that change to make ServerSocketAppender work again is an option or the best idea. Ideally it would be great to have a configuration option on ServerSocketAppender to specify the maximum number of connections, though that would mean each ServerSocketAppender would have to manage its own executor service instead of using the one from the context.

In a fork I have added some protected methods to AbstractServerSocketAppender to allow subclasses to override which executor is used for accepting socket connections and for handling each client connection. Whether or not the fix will be to revert the pool size change, this might be useful to users that want to, by subclassing, customize how many concurrent client connections are allowed to read log messages from a ServerSocketAppender without having to create a whole new context class. I'd be happy to clean it up and contribute it if you want.