SuperSteve3D

Last month I interviewed the developers of Rybka Aquarium, a new graphical user interface (GUI) designed for Rybka. This month I’ll talk to the brains behind Rybka itself – Vasik Rajlich and Larry Kaufman. They are currently working on a completely new version, Rybka 3, which has been in development for almost a year. Rybka 3 will not only offer increased playing strength and greater chess knowledge compared to Rybka 2.3.2a, but also new features such as FischerRandom chess (Chess960) and “persistent hash.”

Q: It’s uncommon for two people to work together on the same chess engine. When did you begin collaborating and why?

Larry: Although Vas had already incorporated ideas from my 1999 article in Chess Life on material imbalances into Rybka, our collaboration started around the end of 2006 and I became officially part of the team at the start of 2007. My involvement in computer chess dates back to 1967 when as a student at M.I.T. I was the chess “teacher” and opening book author for MacHack, the first computer to compete in human tournaments. In the late ‘80s and early ‘90s, I was coauthor with Don Dailey of several chess programs, including “RexChess” and “Socrates,” which excelled in various competitions against human grandmasters. I am not a programmer (tic-tac-toe was the limit of my programming skills), but I know quite a bit about computer chess and about how to evaluate chess positions quantitatively. My role in Rybka is quite similar to my role in the programs with Don Dailey; mainly to determine the values for all the terms in the evaluation function and to propose new ones. The big difference is that now we have the computing power to test each new idea, whereas twenty years ago it was mostly guesswork.

Vas: I became familiar with Larry’s work sometime back in 2003, and we started formally working together around January of 2007. Our division of work is quite simple: Larry is responsible for devising evaluation features and balancing their interactions, and I do everything else. Having Larry on the team has been a huge boost, as he has a long background in working with chess features in a quantitative manner and is simply awesome at what he does.

Q: It’s also unusual to see strong players developing a chess engine, yet both of you are international masters. Larry is also an experienced chess teacher. Would you have approached the development of Rybka differently without this background?

Larry: .My role in Rybka is highly dependent on my chess knowledge as an IM, whereas since I joined the team Vas makes little use of his chess skills but rather focuses on the search, which requires only a general understanding of how chess moves are chosen. Even before I joined the team, Vas tended to rely on automated procedures rather than his own opinions of how to evaluate chess positions.

Vas: That’s an interesting and complex question. As a chess player, I never really tried to break the game down into its nuts and bolts. I would basically just try to beat my opponents (smiles). I didn’t start to think about chess in a structured way until my work on Rybka began.

Every computer chess heuristic, whether it’s search or evaluation, has its root in some sort of human chess concept. In its final form it typically looks much different, but there is definitely some link between the way a person thinks about chess and the way he instructs his program to think about chess. Of course, this link is extremely complex.

Q: What about the effect of Rybka and its development on your own chess career?

Larry: It seems to be helping, since I’m having the best results in many years despite my age (sixty), having just won both the Maryland Championship and the U.S. Senior Championship in the same week.

Vas: It’s been ages since I had any chess ambition or even played in a chess tournament. Chess is a tough sport.

Q: You are both very active on the Rybka forum and your participation is clearly important to Rybka users. Does user feedback affect the development of Rybka?

Larry: I would say that maybe once a month or so an idea from the forum proves helpful to my evaluation work.

Vas: Sure, I think so. Some users give us interesting ideas. It’s also good to stay connected and get a general feel for what expert users are thinking. Of course, a good chunk of forum discussions are just a fancy way of goofing off (smiles).

Q: Who are the other members of the Rybka team?

Vas: Iweta Rajlich is our main tester. She uses the latest (private) Rybka version several hours per day as part of her normal chess analysis and is constantly giving me feedback, as well as finding various little bugs and glitches.

Jeroen Noomen is our opening book author. His book is used when Rybka plays in tournaments, and is also included in the Rybka package, giving users a theoretical reference.

A ton could be written about this topic. I’ll just limit myself to a few brief comments.

* Jeroen is really good at what he does; it’s a huge blessing to have him on the team. Jeroen’s tournament record is incredible – in around one hundred tournament games against the top book makers in the world, going through all sorts of incredible complications and home analysis from both teams, we have never had a position that was worse than equal with white or worse than the starting position with black. This is mixed with about a dozen book wins and another couple dozen overpowering advantages.
* Like Larry, Jereon has a long history in the field, going back nearly twenty years. He’s won several dozen top-flight tournaments.
* In computer chess tournament play, opening theory is of huge importance. In general, the stronger the players, the more important opening preparation is, so it’s no surprise that when 3000+ Elo entities square off, you want to win the theoretical debate.
* The opening books provided to users are an underutilized resource. This is something we aim to change. Sure, some books provided with engines are just database dumps and perhaps this is what users have come to expect, but a really good book such as Jeroen’s contains the latest theory from many different sources as well as all sorts of original analysis. A good book is also well-organized and properly propagates its conclusions “up the tree,” so that the user can immediately understand which variations are currently preferred, going all the way up to the first few moves.

We also have two webmasters, Felix and Christoph Kling from Ingelheim, Germany. Christoph designed and implemented the first versions of the website using his own special tools, and Felix has taken it over since then. I’m very happy with the result.

Q: Have you learned anything from the matches between Rybka and various grandmasters that you were able to transform into improvements in Rybka?

Larry: I would say that each match contributes something. Usually it’s something little, maybe worth one or two Elo points, but I think that the most important development from these matches is the new definition of “contempt” and associated ideas to avoid draws that are part of the eval now. I hope that with Rybka 3, when contempt is set, we won’t see people making draws by blockading the position and “passing”; at least it should be much harder.

Vas: This is Larry’s area. Aside from anything Larry picks up, I do think that there is some sort of general benefit in seeing Rybka in action in a different setting.

Q: Does the work on the evaluation improvements in Rybka 3 focus on specific areas or is it an overall revision involving most evaluation terms?

Larry: It’s an overall revision. Not only are most values changed, there are many new terms, maybe the eval is something like twice as big now.

Vas: Larry tends to work very incrementally, one solid but small step at a time. Evaluation work lends itself to this. After more than a year of such steps, and considering that Larry brought a rather different philosophy to the table, almost everything is very different.

Q: What do you consider the most important evaluation improvements in Rybka 3 compared to the previous version?

Larry: The contempt and related terms are very important against humans, even if they only add three to five Elo points against computers. New endgame knowledge is worth several points, revised definition of game phase helped, pawn structure knowledge is dramatically improved, king safety is much better understood, special terms about piece placement relative to other pieces helped, more sophisticated definition of mobility gained some points, defense of the king was much improved, proper use of the king in the endgame was emphasized. The list is almost endless.

Vas: This is mostly for Larry. There isn’t any one big thing – it’s dozens of little things. We’ll probably try to document this in some detail around the time of the Rybka 3 release, as this could be interesting for users.

Q: Based on the improved evaluation and assuming no other changes, how much stronger is Rybka 3 compared to Rybka 2.3.2a?

Larry: This is a tricky question for two reasons. First of all, increased eval slows down the program. If you don’t consider speed, the increase is huge, maybe 100 Elo or more, but this is not fair. When I played recent versions against 2.3.2a MP at bullet chess, I got results in the +50-60 Elo range. Because the search work was midway, I don’t even know if the search on these versions was better or worse than on 2.3.2a, maybe roughly comparable. So this might be a fair estimate for the net eval gain based on self play. I believe that self-play tends to inflate gains mildly compared to playing other engines, so maybe this would drop to +40 against other engines. Against humans there is a further drop, maybe to +30. In general, the more dissimilar the opponents are, the less the rating gain from any given improvement.

Vas: Again, this is mostly for Larry. The evaluation is much better – it’s simply more accurate in almost every type of position, although of course not in every individual position. The features are better and the way the features interact is more scientific and better justified.

Q: What can you tell us about the search improvements?

Vas: It’s hard to say anything concrete here, first because search changes are hard to describe, and second because everything is still being assembled and I haven’t been able yet to see everything in action as a coherent whole.

Generally speaking, the search will be much more efficient – relevant variations will be searched more deeply, while irrelevant variations will be discarded more quickly. I put special emphasis on the tactical strength and on Rybka’s ability to search through murky, irrational positions, as these have always been an area of relative weakness for her.

Q: Rybka running on multi-core/multiprocessor computers is used extensively by strong chess players, correspondence players, opening book authors and last but not least by freestylers. Do you have any results yet indicating how Rybka 3 will scale on multiple CPUs?

Larry: We already know from the Mexico version that scaling was substantially improved in the version used in the final four rounds there. I don’t know if Vas has made further improvement in scaling since then.

Vas: This was a fairly serious priority in the past year; I sank maybe six weeks of development time into it. I don’t have exact scaling figures to quote yet, but Rybka now scales quite well. I am quite sure that she’ll finally be the best-scaling engine.

Q: Do you still recommend a 64-bit operating system such as Windows Vista 64-bit or Windows XP x64 for running Rybka?

Larry: Using a 64-bit OS is equivalent (for Rybka, not for most other engines) to using a faster processor, by close to a 1.7 to 1 ratio based on the average of reported results on different systems. This should be worth close to 50 Elo points in self play, somewhat less against other engines and against humans.

Vas: Rybka runs around 1.6x to 1.7x faster on a 64-bit operating system than on a 32-bit operating system. Upgrading from a 32-bit operating system to a 64-bit system is easily the cheapest way to improve your computer’s performance.

Q: Rybka offers unique features not available in other chess engines, such as the Randomizer. Are you planning changes to the randomization approach in Rybka 3?

Vas: There will be a little bit of cleaning up and new interface support – without this, the feature is nearly unusable for typical users. However, there will be no major conceptual changes. For Rybka 4, we have some plans to experiment with embedding some sort of Monte-Carlo-type simulations into the search itself. Users will probably see some work in this area over the next two to three years.

Q: Persistent hash is a highly anticipated feature of Rybka 3. What exactly is “persistent hash”?

Vas: A persistent hash is basically permanent program memory. This will allow Rybka to remember her analysis from previous sessions. We have quite a number of plans for this, allowing users to share hashes and manipulate their contents in various ways, etc.

Q: The “set position” command from the UCI protocol allows you to manually set the value for a position in the hash table so that Rybka always uses it instead of its own evaluation. Are you planning to implement this feature in Rybka 3, in connection with persistent hash?

Vas: Yes. This is a few minutes of work on the engine side. The main question is if interfaces will support it and if users will find it interesting.

Q: What are the practical applications of persistent hash for users?

Vas: The persistent hash simply allows the engine to remember its previous analysis. This has the nice property that it can never do any harm, so aside from being useful as an analysis feature, it can be used by book authors for positions at that point in opening theory where the opening is ending and the middlegame is beginning, where they are not sure enough of their choices to commit to them in their books.

There are many interesting and complicated issues that I won’t go into here. The archives in the Rybka forum have more detailed discussions.

Q: Rybka Aquarium stores the results of infinite analysis on disk. It also offers Deep Analysis (DA), which analyzes a specific position deeply. DA can look up and reuse previously run infinite analysis. All DA results are stored in a tree, so you can stop DA any time you like and resume it later from the point where you left off or any other point in the tree. Users can exchange analysis and different DA trees can be merged. Opening books can be extended with these trees. This could almost be a description of the possibilities offered by persistent hash, which will clearly fit nicely into this picture and further enhance the value of DA.

Vas: I haven’t studied this feature in detail; this is probably a better question for the Convekta developers. On the surface, there seems to be considerable overlap between this and persistent hash, but I am sure that Deep Analysis offers additional value as well.

Q: There were frequent releases of Rybka in the 2.x series. Why have you decided to switch to only major releases?

Vas: Well, you could flip that question and ask why there were so many 1.x and 2.x releases. We were new and catered to the ultra-serious engine users, who wanted above all pure strength and the latest of everything and were willing to live with some rough edges. In addition, we used these many releases and the testing that the expert users were happy to do for us to hammer Rybka into shape. All of this allowed us to go forward as quickly as possible.

Now, we’re shifting into a more professional mode. Each release will be tested and polished, and non-expert users should have a program that they are fully comfortable with.

Q: Rybka WinFinder is perhaps the most tactical chess engine currently available. Can we expect something similar for Rybka 3?

Larry: Yes, there will be some options for more/less speculative evaluation, as well as (I think) one or more search options. Vas will have to speak to the search part of the question.

Vas: I spent a lot of time working on the search around the time of Rybka 2.2, and the WinFinder was a sort of experiment with some of the wilder ideas. I spent only about two days on the WinFinder itself and it’s a bit unpolished.

The WinFinder concept is quite good and if there is time, we’ll have a Rybka 3 WinFinder as well. This tool could be really nice if I were to spend something like two weeks on it.

Q: Are you planning to release Rybka versions for platforms other than Windows?

Vas: Yes. As soon as Rybka 3 for Windows is out, I’ll sit down and make a plan for this.

Q: You mentioned in a recent post on the Rybka forum that your top priority is keeping Rybka the strongest available chess engine, but you have also considered diversifying and going in other directions. Could you tell us more about the options that you are considering?

Larry: We are already adding options so that the user can choose a style more to his liking, as well as FRC support. Other forms of chess or even other games are also possible.

Vas: The things we’ve learned in developing Rybka would be really useful in a bunch of other domains, so it’s only logical to take advantage of this at some point. We’ll need to pick the right moment for it, as chess is very interesting and definitely not something we’d want to lose. There are tons of ideas for other directions. One logical first step would be Shogi.

Q: The Rybka forum is abuzz with anticipation as to when Rybka 3 will be released. When will you put an end to this speculation?

Larry: Since my evaluation work is continuous and incremental, one date is as good as another for me. This is not the case with Vas.

Vas: Sometime before Rybka 3 is released (smiles). Seriously, almost all of the mistakes we made in the past were from rushed schedules. I’d like to see what a different set of mistakes looks like (smiles).