This thesis deals with the instabilities arising in the boundary layer connecting an accretion disk, resulting from mass transfer within a binary star system, and the underlying mass gaining star. I examine the role of critical layer shear flows and the Miles instability as a mixing agent. The specific application is to the onset of the thermonuclear runaway (TNR) on accreting white dwarf stars (WD) in classical novae in which hydrogen and helium-rich matter accumulated above a C/O or O/Ne rich envelope. The increased mixing between the two layers, e.g., Calder et al. (2002), increases the H mass in the ejecta of some observed nova outbursts above the deep mixing expected from turbulence and its driving by buoyancy above the deep mixing resulting from strong convection, beyond the deep mixing produced by Kelvin-Helmholtz vorticity, and its cascade to turbulence during the strongly convective nuclear burning stage (e.g., José et al. (2020)). I applied the mechanism of critical layer instability (CLI) that has previously been used in oceanography (wind-generated waves) by Miles (1957), Phillips (1958), Phillips (1960), to the discontinuity between the accreted H/He and C/O or O/Ne rich layer treating the accreting atmosphere like a wind, and the WD envelope as the sea surface. The resonant interaction between a large-scale shear flow in the accreted envelope drives interfacial gravity waves. The greater compositional buoyancy in the C/O WD means that the interface sustains gravity waves. With a shear flow (that is, a wind), those waves with a group velocity matching a phase velocity in the shear layer are resonantly amplified. These waves eventually form cusps, break, and inject, like ocean waves, a spray of C/O into the H/He layer. However a similar scenario should be evaluated in the boundary layer between accretion disk and a compact star, or, wherever the mixing is expected to play an important role. In chapter 2we reviewthe analytical results of Alexakis et al. (2004) using theWKBJ approximation a physical interpretation following Lighthill (1962), and include a discussion of the physical significance of the results. In the chapter 3 we present simulations of the accretion environment using the PLUTO hydrodynamics code (Mignone et al. (2007)), making the comparison with the results of Calder et al. (2002) of the same scenario using the FLASH code (Fryxell et al. (2000)). The principle results and their application to the classical nova nuclear ignition problem are described in chapter 4: the TNR will trigger in a pre-mixed layer. This forms as a purely hydrodynamic effect (without classical convective driving) and modifies the initial conditions used in current models of the explosion energetics and dynamics. The effect of residual mixing on recurrence timescale is briefly discussed. Pre mixing may be, in part, responsible for this distinct class of novae. I also discuss implications of the simulations for the more general problem of accretion disks in stellar binary systems with non-degenerate gainers (e.g., Algol and related system).