An open system is one in which matter or energy is transfered with other systems. It contrasts with a closed system which is isolated and has no interaction, physical or meta-physical, outside itself.
Contemporary mindsets tend to see discrete divisions in organisation and social structure, like differences between families, companies, or nations. That is reasonable given the apparent separation between objects, including people. However, the reality is that boundaries are permeable and interactions occur between systems to a greater or lesser degree.
For example, a car appears to be be a closed system, but it is not one because it takes fuel from the pump and requires water and air, and it releases heat, motion and pollution all of which affect other systems around it - the biosphere. Even the biosphere is not entirely closed because it takes energy from the sun and radiates heat.
It is useful to treat some systems as closed in order to understand them or because they appear closed in a particular context, like the biosphere - it is wrong to think that pollution will go out of the biosphere in to space.
Traditional mindsets see individuals and social organisations, from companies to nations, as discrete entities operating independently. That mindset served the purpose of focussing attention on particular goals, like winning the war.
Modern analysis, however, recognises interdependence between systems. At the most basic level it is now known that everything is composed of the same subatomic particles which follow univeral laws. At the human level, we know that the behaviour of any person or organisation, such as a business or nation, affects those around it.
Open systems are becoming foundation understanding in many areas. Here are some links.
Wikipedia: Open system (systems theory)
Wikipedia: Open system (computing)
Wikipedia: Open-system environment reference model
ReferenceForBusiness: Open and Closed Systems
Open business refers to a management style in which more autonomy is allowed throughout the organisation. It eschews old ideas of hierarchy which reflect medieval or feudal thinking and adopts structures and systems which enable individuals to adapt to changing situations without persmission from the CEO. The structure is flatter and operations more efficient. It tends to offer guidelines rather than rules and trust rather than control and is appropriate for a modern world of educated people willing to take initiative and responsibility and who aim for common goals.
Many of us use open systems everyday without realising it: The internet and corporate servers perdominantly use unix based open source software. Open source software allows its code to be accessible by anyone. This has been useful in catalysing its development and adaptation to other software and computer systems, making it stable and secure and therefore the preferred software for mission critical applications (like web servers and corporate servers).
In a business context, an open space meeting offers a good experience of open systems. The approach seems antithetical to business because it does not perordain an outcome, but is initiated with barely any agenda but seeks to artculate problems, stimulate alternatives and agree outcomes in real time. Nevertheless, empirical results show that open space meetings resolve issues far more quickly and with fewer resources than traditional approaches.
Wikipedia: Open Space Technology