Plan Merging in Multi-Agent Systems. Mathijs de Weerdt

Transcriptie

1 Plan Merging in Multi-Agent Systems Mathijs de Weerdt

2 Illustrations: Welmoed Kreb Cover illustration: Welmoed Kreb Cover design: Joke Herstel, Wenk

3 Plan Merging in Multi-Agent Systems Proefschrift ter verkrijging van de graad van doctor aan de Technische Universiteit Delft, op gezag van de Rector Magnificus prof.dr.ir. J.T. Fokkema, voorzitter van het College van Promoties, in het openbaar te verdedigen op maandag 15 december 2003 om 15:30 uur door Mathijs Michiel DE WEERDT doctorandus in de Informatica geboren te Alkmaar

4 Dit proefschrift is goedgekeurd door de promotoren: Prof.dr.ir. H.J. Sips Prof.dr. J-J.Ch. Meyer Samenstelling promotiecommissie: Rector Magnificus Prof.dr.ir. H.J. Sips Prof.dr. J-J.Ch. Meyer Dr. C. Witteveen Prof.dr.ir. P.H.L. Bovy Prof.dr.ir. R.E.C.M. van der Heijden Prof.dr. C. Roos Dr. K.S. Decker Dr.ir. N. Roos voorzitter Technische Universiteit Delft, promotor Universiteit Utrecht, promotor Technische Universiteit Delft, toegevoegd promotor Technische Universiteit Delft Katholieke Universiteit Nijmegen Technische Universiteit Delft, reservelid University of Delaware, VS Universiteit Maastricht TRAIL-Thesis Series T2003/8, The Netherlands TRAIL Research School Published and distributed by: Mathijs M. de Weerdt ISBN: Keywords: multi-agent systems, AI planning, coordination, resource allocation Copyright c 2003 by Mathijs M. de Weerdt All rights reserved. No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without written permission of the author. Printed in The Netherlands

5 Preface The research presented in this thesis was performed at the Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS), formerly called Information Technology and Systems (ITS) of the Delft University of Technology. Within the Parallel and Distributed Systems (PDS) group, a project on Collective Agent-Based Systems (CABS) was started in The CABS program aims at developing specification methods, algorithmic techniques, and implementations of large-scale agent-based systems and organizations of agent-based systems. Since these formalizations and algorithms can contribute to the solution of scientific, business and societal problems in the field of transport, infrastructure and logistics, this research is carried out within the research school on Transport, Infrastructure and Logistics (TRAIL, 1994 ), more specifically within the Delft interdisciplinary research center (DIOC) research program Seamless Multi-Modal Mobility (SMM). The Collective Agent-Based Systems (CABS) projects The research projects within the CABS project concern two different themes. On the one hand, a theory and methods are being developed to realize inter-organizational coordination. On the other hand, incident management methods are studied to guarantee the execution of coordinated plans in dynamic situations. This work on incident management is carried out by two colleagues. Jonne Zutt (2000) concentrates on the recognition of unexpected events (diagnosis) and the reactions on an operational level, whereas Roman van der Krogt (2000; 2003) is more interested in how a whole plan should be modified on a tactical level to deal with more serious events that affect the applicability of such a plan. The research on inter-organizational coordination in the CABS project focuses around the coordination of the planning of actions. Initially, a distinction is made between preand post-planning coordination processes. Jeroen Valk (2001) studies multi-agent task allocation protocols to guarantee that each agent is able to construct its own plans autonomously (pre-planning), while the work in this thesis shows how to coordinate such separately constructed plans afterwards (post-planning). Eventually, the planning and the coordination processes should be integrated into a single coordinated planning process. i

6 ii Plan Merging in Multi-Agent Systems Finally, the theme on inter-organizational coordination is supported by the development of a specification language for coordination processes in multi-agent systems. This work uses a combination of game theory and (modal) logic, and is carried out in cooperation with the Intelligent Systems group at Utrecht University (Harrenstein et al., 2001). The first main contribution of this thesis is a formal framework to express multi-agent planning problems and multi-agent plans in an object oriented way. This framework can also be used on the operational level to formulate the replanning of actions. A second contribution is a method to improve multi-agent plans. This method is formally introduced and analyzed as well as subjected to an empirical investigation using a data set from a taxi company. These techniques for solving multi-agent or multi-organizational planning and coordination problems in general will help to solve specific coordination problems in many domains, and not in the least, in transportation domains. Acknowledgments I would like to thank everyone who supported me while I was writing this thesis. First, and foremost, I thank Cees Witteveen, who was and is always available to guide and encourage me. Second, I thank John-Jules Meyer for his enthusiasm, and Henk Sips for trying to speed up my finishing this thesis. All of them always combined their constructive comments with stimulating statements. Furthermore, I thank Nomi Olsthoorn, who not only helped me with her favorite statistical tool, but also supported me upon drawbacks and made me aware of my luck (especially compared to hers). I am grateful to Roman van der Krogt, because he almost always offered useful critique and a cup of tea whenever I was stuck. I thank Andr Bos and Hans Tonino for their ideas and support in the first couple of years of my research, and Jonne Zutt, Jeroen Valk, and Paul Harrenstein for their easy company at every CABS occasion. I am also very grateful to all my other colleagues in the Parallel and Distributed Systems group (Arjan, Dick, Frits, Jan, Johan, Kees, Koen, Leo, Linda, Marcel, to name the most important ones) for creating a friendly and healthy (well, maybe except for the cake on Friday) working environment, and especially Leon Aronson for proof-reading most of my thesis. Besides, I am quite happy that Ard Biesheuvel and Lon Planken finished the CABS-planner before I finished my thesis. I appreciated all the (social) events, courses and all kinds of forms industriously provided by TRAIL, and I thank Piet Bovy and Ruud Sommerhalder for making this research school, the SMM project, and more specifically my own research within CABS possible. I thank Wiebe van der Hoek for pointing out the CABS research program and driving me to Delft for my application visit. More concrete results of people helping and supporting me are the data set used, which

7 Preface iii is provided by Pim van der Stoel from Tens B.V. and Ron Kooi from Taxi Zeevang, the help I received from Mirjam Nieman to improve my style of writing, and the illustrations about the planners Pete, Monica and Ralph transforming to a (merged) taxi made by Welmoed Kreb. I would also like to thank Robbert Kok for his warm welcome in Delft and together with Victor van Rijn and Hiske Wegman for their friendship and inspiring discussions. I thank my parents for their stimulating interest and encouragement, my brother Maarten for his very good friendship ever since I left home and Barbra for her support when Maarten had a heart operation and my parents were in Ecuador. Finally, I am very happy with the sea scouts from Marco Polo Alkmaar and all my friends there, because whenever I am among them I completely forget all about this research and its unsolved problems.

8 iv Plan Merging in Multi-Agent Systems

9 Contents 1 Introduction Inter-organizational coordination Research body 3 3 Conclusions and perspectives Conclusions Future work Extensions of the formalism Extensions of the plan merging algorithm Further experiments More advanced multi-agent planning algorithms A Mathematical notations and definitions 11 Bibliography 13 Index 15 Summary 17 Samenvatting 21 Curriculum Vitae 25 v

10 vi Plan Merging in Multi-Agent Systems

11 Chapter 1 Introduction 1.1 Inter-organizational coordination Somewhere in a medium to large city, Pete is working in the office of a moderately small taxi company, called P-Tax. Pete is the planner of this company. His job is to assign transportation tasks to the taxi-cabs. First, he enters transportation requests from customers into a database in his computer. Then a planning algorithm composes a possible schedule. Pete is glad to have such help, because especially planning the shared transportation in vans, also offered by this company, can become quite complicated. During the day new requests come in, and the schedule is updated regularly. Sometimes, when the number of requests is extraordinarily high, Pete contacts Monica, a befriended planner at another taxi company (M-Cabs) by phone, and asks for help. Usually, M-Cabs is able to take over some of the requests. Once or twice a day a customer requests an unusually long trip. If the cab concerned has to return empty, this is a very inefficient use of resources. Therefore, whenever he can, Pete checks with Monica to see if they have a trip in that direction as well, or even better, a trip on the way back. If possible, the planners agree on combining their requests. A similar agreement is sometimes made when passengers need to be picked up in a region or suburb where one company has some cars available nearby, while the cabs of the other are at the other side of town. Both taxi companies P-Tax and M-Cabs profit from such cooperation. 1

12 2 Plan Merging in Multi-Agent Systems

13 Chapter 2 Research body See Example 2.1 for an interesting example. Example 2.1. An example of a resource fact where terms are variables or functions on terms is the resource fact taxi 2 (p : plate,l : loc,3 c : capacity) that specifies a taxi at an unspecified location l with capacity (3 c). Such resource facts are very useful to define goals and conditions for actions. Note that this resource fact has identity value 2. To formally define terms and such resource facts, we first introduce all symbols that can be used in our language. As said before, a formula in RL is typically used to describe a state of the world by a conjunction of resource facts. Each resource fact can have several attributes of different sorts. So we need symbols for conjunctions, sorts, variables, constants, and the resource predicates. Theorem 2.2. (Haslum and Jonsson, 1999) Complexity of conformant planning. Deciding the existence of a conformant plan (sequence) for a problem Π with an unobservable propositional domain and actions is EXPSPACE-complete. 1 In a multi-agent system, agents often perform actions unexpectedly and independently of each other. When each agent is planning on its own without communicating and coordinating with the other agents, each agent has to solve some sort of non-deterministic planning problem. Theorem 2.2 and the complexity analysis of contingent planning show that non-determinism in planning is EXPTIME-hard. So we can conclude that such an individual approach 1 The class EXPSPACE is the class of decision problems that can be solved using an amount of space bounded by 2 p(n), where p is a polynomial and n is the input length. It is known that the class EXPTIME EXPSPACE contains problems that are intractable (Garey and Johnson, 1979, Theorem 7.16 in), even if P=NP. 3

14 4 Plan Merging in Multi-Agent Systems G unload(passgr) move(b,c) load(passgr) move(a,b) I Legend a() I G action set of initial states set of goal states Figure 2.1: A sequence of actions (plan) leading from each of the initial states in I to one of the goal states in G.

15 Chapter 3 Conclusions and perspectives The aim of this thesis was to provide a problem definition, a framework, and methods for multi-agent planning. These three contributions should help to improve techniques for inter-organizational coordination. For example, the taxi companies P-Tax and M- Cabs like to have a coordinated planning system to take advantage of situations where coordination can lead to a significant reduction in the operational costs. It is the year Both taxi companies P-Tax and M-Cabs have been running the new version of the taxi planning and scheduling software (TPSS) with integrated coordination modules for more than a year now. Recently, the (now) managers Pete and Monica have signed a coordination agreement to formally define how to split the revenues of sharing and exchanging rides. The new planning personnel at the office is reviewing the automatically generated schedules. Their main task is to keep in touch with the taxi drivers executing orders to deal with unexpected events and to explicitly approve (or disallow) the execution of a coordinated ride. Furthermore, they provide the TPSS with up-to-date information on the availability of taxi drivers, road blocks, queues, the weather, special events, etc., and they tune the parameters of the TPSS such as the slack on the drive times, and the time margins for picking up passengers. Multi-agent planning techniques are especially useful to coordinate plans and processes between organizations. However, further research on the multi-agent planning problem and development of distributed planning heuristics will also enable the use of these techniques in situations where the autonomy and privacy of the agents is a less important factor, and where currently centralistic planning and scheduling techniques are used. For example, the plans of subdivisions of a company, of departments of a hospital, or of divisions of an army may be more efficiently approached by multi-agent planning techniques. 5

16 6 Plan Merging in Multi-Agent Systems In general, whenever a problem can naturally be decomposed into related subproblems, it can be modeled and solved as a multi-agent problem. See Algorithm 3.1 for more info. Algorithm 3.1. SOMETHING SMART 1. do this 2. and do that 3. for my sake do (a) do a lot of this 4. print results 3.1 Conclusions Most multi-agent planning research focuses on a specific practical instance of the multiagent planning problem. In this thesis, multi-agent planning was formally defined as the combination of the planning problems of a set of agents in the same environment. The formal problem turned out to be strong and flexible enough to model a realistic multiagent planning problem in the taxi scheduling domain. Multi-agent planning was proved to be an extremely difficult problem, i.e., PSPACE-complete. This problem definition can be used to give more structure to the process of comparing different approaches to multi-agent planning. In existing frameworks to describe planning algorithms, a state of the world is described by an unstructured set of properties of all objects relevant for the planning problem. Especially for multi-agent planning problems, properties of objects should be modeled in a more object-oriented way. The framework introduced in this thesis consists of a subset of linear logic on so-called resource facts and an operational semantics. The resource facts represent resources or products and describe all the properties (attributes) of such an object at a specific time. Actions are rules to transform a set of resource facts. To model more complicated domains, the formalism supports the use of constraints on the attributes of a resource fact to impose restrictions on the execution of an action. Using resources as the atomic objects of a planning problem and actions as resource-consuming and resource-producing processes appeared to work very well in a practical multi-agent planning problem. The resource language was used to derive that an action can be removed from a (multiagent) plan while maintaining the realizability of all goals (of all agents) if for each used output resource a replacement can be found. Moreover, this condition was used to design

17 Chapter 3. Conclusions and perspectives 7 a polynomial any-time approximation algorithm for the plan merging problem. The plan merging problem was defined to be a search problem of finding available resource facts to satisfy a given set of requests for removing one or more actions. This problem was proved to be NP-complete, which is plausibly less difficult than the multi-agent planning problem. 1 A ground version and a more flexible version of the plan merging algorithm were proved to have a time complexity of O(n 2 ) and O(n 1+c ), respectively, where c denoted the maximal size of an action. The properties of the ground plan merging algorithm were analyzed by doing experiments using real data from a taxi company. A schedule for 35 taxis was used to create a hypothetical experiment. In this experiment we divided the taxis randomly over a set of artificial taxi companies (i.e., the agents). The plans of these agents were then merged using the plan merging algorithm. Evidence was gathered showing that the ground plan merging algorithm had a very good any-time behavior and a quadratic time complexity with a very low constant. Further experiments showed that when the taxi company would allow passengers to share a taxi, the reduction of the costs of a plan were strongly related to the additional travel time of passengers. For example, for large plans and a pick-up and drop-off margin of at most 6 minutes, an increase of 11% in travel time was shown to result in a distance reduction of about 13%. To round off, we conclude that both the formalism and the algorithm work quite well on realistic data and we believe that the proposed problem definition and the formalism are useful for further research on coordinated planning. 3.2 Future work This thesis attempted to find general methods to solve instances of the multi-agent planning problem. The solid foundation defined here will help to formulate subsequent research aims, and hopefully also to describe new research approaches. Further study includes extending the formalism, improving the plan merging algorithm, doing more experiments, and, last but not least, developing more advanced coordinated planning methods. Each of these issues is detailed in the following subsections Extensions of the formalism The action resource formalism (ARF) is quite powerful, but some planning problems and algorithms have aspects that cannot be modeled explicitly in the ARF. For example, sometimes the duration of an action is relevant. This may be modeled by including time 1 It is assumed that NP PSPACE (Garey and Johnson, 1979).

18 8 Plan Merging in Multi-Agent Systems attributes in all consumed and produced resources, and by parameterizing the state of the world (during the execution of the action) by a time variable t. The attributes of some (produced) resource can then be described by a function of t. In some applications the non-existence of resources is required, e.g., a goal could be I do not want any passengers. The semantics of this goal is that for the resulting plan P for any attributes x 1,...,x n the resource passgr(x 1,...,x n ) Out(P). Such a requirement may be translated to constraints on the goals and the actions, or it should be defined in the operational semantics. Furthermore, sometimes it is necessary to define that a resource should not occur in any state during the execution of a plan. This is an invariant over a plan about a resource that should not be produced anywhere in the plan. A third example of a possible required extension of the ARF is when we need to be able to specify in the problem description that we would like to have as much of something as possible. This may be modeled either by specifying that we need as much of the resource (e.g., money) as possible, or that the attribute amount of this resource should be as large as possible. In some multi-agent situations, agents may need to include a joint action, such as joining a meeting or handing over a valuable package. In the ARF we have no method to describe actions that require the direct involvement of multiple agents, other than defining an action in one of the agents plans that needs resources from the other agents. If an application domain requires one of the above issues, we need to thoroughly investigate the best way to extend the formalism Extensions of the plan merging algorithm The plan merging algorithm may also need a couple of extensions. In many domains, agents can only benefit from the exchange of a set of resources. A combinatorial auction (Sandholm, 2002, e.g., ) is required to deal with these kind of exchanges. This extension also requires a formal analysis of plan reduction with more than one action. Another interesting extension is to auction resources to private individuals as well, e.g., via ebay (2003), or in the transportation domain via specialized websites. And then, especially when private individuals are involved, the algorithm should be able to deal with unfair and selfish agents (Ronen, 2000). To use the plan merging algorithm in a somewhat more dynamic environment, we need the functionality of adding requests during execution. Such an extension is a step towards more advanced multi-agent planning algorithms. However, first, the plan merging algorithm should be tested on other data sets and in more domains.

19 Chapter 3. Conclusions and perspectives Further experiments The experiments presented were based on a couple of strong assumptions. We need to check the validity of these assumptions by doing plan merging on this data while taking into account (i) breaks of taxi drivers, (ii) the long-term effects of picking up passengers on the rest of the plan, and (iii) an estimation of the travel time that includes the distance on the road map, the number of traffic lights, and queuing time based on the time of the day. We also need to check that there is no communication overhead. Furthermore, the assumption has been made that the artificial set-up of the experiment is realistic (the centralistic planning from one company was used to derive plans for a whole set of much smaller taxi companies). If information from several companies in the same region and from the same days is available, this assumption can be verified. Such information can also help to test different award models for companies that help others. The conclusions on the plan merging algorithm can be made even stronger if the algorithm is run on other data sets, including those from other taxi companies, but preferably also data sets from other domains, such as transportation scheduling, pizza delivery, hospital scheduling, computer games, robot rescue teams, army mission planning, or controlling cooperating satellites. It is also useful if simulation data for these domains can be generated, because then they may be used as benchmarks. Benchmarks help to compare different plan merging algorithms, or even other multi-agent planning methods More advanced multi-agent planning algorithms The most important further study should be on the integration of the coordination in the planning process itself. If agents can communicate and adjust their plans while these are under construction, conflicts can be solved more elegantly, and agents that do not have enough resources themselves might still be able to construct a plan. An initial proposal for such a coordinated planning system has already been made (De Weerdt and van der Krogt, 2002), based on a forward heuristic. In this thesis the basis for such an extension was laid out by the introduction of a formalism and algorithms to coordinate plans after they have been formed. Several existing planning techniques such as task reduction planning and partial order planning can, in principle, also be used in a coordinated planning system. Each technique requires different modifications of the multi-agent planning system. For example, for partial order planning we need to describe some additional conditions on the existence of resources (for example during an interval (IPC). For other domains, task reduction planning methods seem more appropriate. In that case we need at least subplans, an abstraction of plans, such that they look just like actions, and a plan library (Van der Krogt et al., 2003).

20 10 Plan Merging in Multi-Agent Systems Specialized, domain-specific optimization algorithms for subproblems usually perform better than a general (planning) problem solver, for example a shortest path algorithm is much faster than using a planner to determine a shortest route in a graph. It is therefore valuable that such algorithms can easily be incorporated in a multi-agent planning system, for example as lower and upper bounds. This way one sometimes can still have the ease of having a method that can be applied in many domains, while also being able to take advantage of long lines of specialized problem-solving research. Probably, however, this is not always possible. Finally, for many real applications, the execution of a plan needs to start even before the plan has been completed. Consequently, also for multi-agent planning we need to look into methods to do planning and execution simultaneously.

21 Appendix A Mathematical notations and definitions Some other relevant information... 11

22 12 Plan Merging in Multi-Agent Systems

23 Bibliography de Weerdt, M. M. and van der Krogt, R. P. (2002). A method to integrate planning and coordination. In Brenner, M. and desjardins, M., editors, Planning with and for Multiagent Systems, number WS in AAAI Technical Report, pages AAAI Press, Menlo Park, CA. ebay (2003). ebay - The World s Online Marketplace. Garey, M. and Johnson, D. (1979). Computers and intractability a guide to the theory of NP-completeness. W.H. Freeman and company, New York, NY. Harrenstein, P., van der Hoek, W., Meyer, J.-J., and Witteveen, C. (2001). Boolean games. In Proceedings of the Eighth Conference on Theoretical Aspects of Rationality and Knowledge, pages Morgan Kaufmann Publishers, San Mateo, CA. Haslum, P. and Jonsson, P. (1999). Some results on the complexity of planning with incomplete information. In Proceedings of the Fifth European Conference on Planning (ECP-99), volume 1809 of Lecture Notes on Artificial Intelligence, pages Springer Verlag, Berlin. Ronen, A. (2000). Solving Optimization Problems among Selfish Agents. Ph.D. thesis, Hebrew University of Jerusalem. Sandholm, T. W. (2002). Algorithm for optimal winner determination in combinatorial auctions. Artificial Intelligence, 135(1 2):1 54. TRAIL (1994 ). TRAIL mission statement: Research school for transport, infrastructure, and logistics. Valk, J. M., Witteveen, C., and Zutt, J. (2001). Approximation results for multi-agent planning systems. In Proceedings of the fourth Pacific Rim International Workshop on Multi-Agents (PRIMA), pages ACM Press, New York, NY. van der Krogt, R. P., Bos, A., de Weerdt, M. M., and Witteveen, C. (2000). An algorithm for replanning. In Proceedings of the Twelfth Belgium-Netherlands Artificial Intelligence Conference (BNAIC-00), pages

24 14 Plan Merging in Multi-Agent Systems van der Krogt, R. P., de Weerdt, M. M., and Witteveen, C. (2003). A resource based framework for planning and replanning. In Proceedings of the 2003 IEE/WIC International Conference on Intelligent Agent Technology (IAT-03). Zutt, J. (2000). Transport Control in a Multi-Agent Setting. Master s thesis, Delft University of Technology.

25 Index CABS, i complexity of conformant planning, 3 EXPSPACE-complete, 3 symbols, 3 15

26 16 Plan Merging in Multi-Agent Systems

27 Summary Plan Merging in Multi-Agent Systems By coordinating their plans extra carefully, organizations may be able to come up with activities that do not conflict and whose resources are used more efficiently. This is very important when coordinating logistic processes, such as the planning of routes for taxi companies and the efficient allocation of transportation orders to trucks for transport companies. However, also outside the transportation sector, operational costs can be significantly reduced by better cooperation. Think for example of the coordination of several armies during a UN peace mission or of the coordination of different departments within a hospital. The coordination of activities planned by several autonomous companies is a very important aspect of inter-organizational coordination. Most organizations, however, do not want to relinquish all information concerning their private plans, as they are afraid of losing their autonomy. Hence, in general, organizational coordination problems cannot be solved centrally. Therefore, we have to search for coordination methods that are able to coordinate individual plans in a distributed way, while respecting the privacy of the individual planning agents (companies, people). This is exactly the topic of multi-agent planning, where methods are developed to solve a set of related (single-agent) planning problems. In this thesis we develop a resource-based framework to model and solve such multiagent planning problems. In this Action Resource Framework (ARF), resource facts are taken as the central concepts for both planning and coordination of actions. They are used to give an exact description of the primitive objects and their properties in a planning domain. In the taxi domain, these resource facts are for example the taxis, the drivers, and the passengers. Actions in this framework are processes that consume and produce such resource facts, e.g., the movement of a taxi, or passengers getting in or out. In general, an action has a precondition to denote the required input resource facts, and a description of the output resource facts produced by the action. Goals are specified by a formula that describes the set of resource facts that has to be obtained in order to satisfy the goal. A description of a goal for a taxi company is, for example, a description of a passenger at a certain location at a certain time. To attain 17